Genome Analysis Wiki - User contributions [en]

SardiNIA

2016-07-12T19:51:45Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

== Project Leaders ==

* David Schlessinger (National Institutes on Aging, Baltimore)
* Manuela Uda (National Research Center, Cagliari, Italy)
* Goncalo Abecasis (University of Michigan, Ann Arbor)

== Genomewide Association Study ==

We carried out an initial genomewide association by genotyping 1405 samples with Affymetrix 500K SNP arrays. Then, because SardiNIA samples are closely related to each other, we were able to use these genotypes to impute the genomes of many close relatives who were genotyped with Affymetrix 10K SNP arrays. Typically, our SardiNIA GWAS analysis thus include approximately 4300 genotyped or imputed individuals. To learn more about the approach see Chen and Abecasis (2007) and Scuteri et al (2007).

=== Planned Updates ===

Several ongoing experiments are expected to gradually improve our GWAS data. First, we expect to integrate Affymetrix 1M SNP chip genotypes into our analyses. Second, we plan to genotype all sampled individuals with the Metabochip, which includes 200,000 SNPs. This will enable us to fill in genotypes for relatives of those genotyped with denser arrays more accurately, because we will more precisely identify shared stretches of chromosome.

== Medical Sequencing Project ==

We are sequencing the genomes of 1,000 individuals to learn about the genetics of blood lipid levels and personality.

== Status as of July 2016 ==

=== Locations of Files for Current Data Freeze of 3839 Samples===

NOTICE: We identified late in the process that two of the samples (22855 and 22358) were actually the same individual. They both should be the same individual 22855. Therefore, there are 3840 sample IDs in each of the files below, but only 22855 should move on to later processes. In future data freezes with this data, these two sequencing sets should be merged into a single 22855 individual.

* List of '''Sample Numbers'''
** The following file contains three columns: [SampleID used in these analyses] [ID supplied by CSCT or Sardinia or other project] [Sequencing core ID (if different)]:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/sampleIDConversion.txt

* List of paths to '''BAMs''' used in this data freeze (Index file)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_index_20150504.index
** 401 of these samples have some new BAM contribution since the previous data freeze... their BAMs can be found here: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/newsamples.index

* '''Pedigree''' (Not too helpful -- used for SNPCall)
** All Samples: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_ped_20150510.ped
** Disjoint Trios: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/Pedigree_Fall15_DataFreeze_Triplets.ped

* '''QC''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCStats.txt
** For a list of paths to all of the separate QPLOT files for each sample, see the file: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCFileListFinal.txt

* '''SNPCall''' Results (Produced with Gotcloud SNPCall and phased using Beagle4)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''IndelCall''' Results (Produced with Gotcloud Indel)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/indel/final/all.genotypes.sites.vcf.gz
** SNPEff and VEP declarations of Indel types can be found:
*** SNPEff: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/snpEff/*
*** VEP: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/VEP/*
** We used an '''Indel filtering strategy''' composed of many levels.
**# AC must be 1 or greater -- eliminate Indels with AC=0
**# the Indel should overlap with a VNTR region or overlap with another Indel. We used Adrian's annotate indels program to identify such overlaps. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/All.annotated.sites.vcf.gz
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/Overlaps.txt
**# at least 50% of the Indels should have informative AD field (we define "informative" to mean that the sample has U/(R+A+U)<0.50). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter2_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# at least 50% of the Indels should have informative PL field (we define "informative" to mean that the PL field for the sample is anything BUT ././. or 0/0/0). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter1_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# the Indel needs to have BF_LRE_LUD (a Bayes factor comparing a. related & HWE to b. unrelated & HWD) > -10. Higher BF_LRE_LUD should indicate a better Indel. We used Hyun's MiLK program to obtain BF_LRE_LUD values. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/all.genotypes.milk.sites.vcf
*** Overall results from all of the below filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/FinalIndelFilteringStatistics.txt
*** VCFs of Indels after filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.*.PASS.vcf.gz
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.PASS.vcf.gz

* Merged Indel VCF with beagles SNP VCF, then sorted to get the following VCFS:
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/chr*.PASS.beagled.vcf.gz

* Indel and SNP VCFs that have been combined AND BEAGLED AGAIN TOGETHER using Beagle4. '''These are the latest VCFS''':
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/beagle4/beagle4_chr*/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''mtDNA Copy Number''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/sardiniaCopyNumber_include.txt
** By sample copy number results by sample can be found: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/*.CopyNumber.noRand.500000.*.txt

* '''Filtering Samples''': Discrepancy Figures
** We compared chip data from previous work to the sequencing data produced now in hopes of identifying which samples have reliable sequencing data. This comparison was done for each chromosome separately and then combined into an overall discrepancy figure. Only 3189 of the 3840 samples (3188 of 3839 if you throw out sample with two sets of data) had chip data.
** By chromosome discrepancy figures for each sample can be found:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/*.diff.discordance_matrix
** By chromosome discrepancy figures for all samples in one file can be found:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/all.discordance.summary.txt

=== What is Complete ===
* SNP Call
** 24,901,469 SNPs passed filters
*** 16,822,922 are in dbSNP (67.6%)
*** %Known Ts/Tv - 2.24
*** %Novel Ts/Tv - 1.95
* InDel Call
** 1,194,945 passed filters

=== Future Directions ===
* NOTE: For sample filtering below, you will need to finish chromosome 1 for me. I have it currently running. Once it is done in a few days, you will need to run the command 'python calculateConcordance_onefile.py' while in the '/net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr1/' directory.
* '''Sample Filtering'''
** We did not do any filtering of samples (based on dupRate, genome coverage, mapping rate, proper paired, mean depth, or any other QPLOT stats) prior to SNP and Indel calling. Because of this, we want to do this filtering now. 3,188 or 3,839 samples have genome chip data from a few years ago. For these, we could look at the non-reference concordance between the chip genotypes and the sequencing genotypes and declare 'bad' samples to be those that fall below a certain threshold, such as 98% non-ref concordance. However, since the remaining 651 samples do not have chip data, this is not an option for them. Therefore, we decided on the following strategy instead:
**# Calculate non-reference concordance for the 3,188 samples that have chip data.
**# Create a prediction model using QPLOT statistics as predictors of non-reference concordance. Either do so on all of the 3,188 samples and look at R^2 (likely inflated from overfitting) or use cross-validation (test and training set) to give a measure of external predictive power.
**# If reasonable predictive power/R^2, use the prediction model to estimate the non-reference concordance amongst the 651 samples that do not have chip data. Also use the prediction model to estimate the non-reference concordance among the 3,188 samples that do have chip data.
**# Set a cut-off for 'good' versus 'bad' samples based on the estimated non-reference concordance and use it to filter samples.
** NOTE: The number of positions for which the chip data give 0/0 and sequencing gives 0/0, chip data gives 0/0 and sequencing gives 0/1, chip data gives 0/0 and sequencing gives 1/1, chip gives 0/1 and sequencing gives 0/0, etc. etc. BY chromosome can be found in the files /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/all.discordance.summary.txt. These can be used to calculate overall non-reference concordance across all chromosomes.
* '''Mitochondrial Depth Analysis'''
* '''Telomere Length Analysis'''
** Investigate the associations between telomere length (an indicator of aging) and variants. Likely interesting in Sardinia population because Sardinians have longer lifespans & centenarians.
* '''Phenotype Study'''
** Likely will not yield much because not many additional samples since Carlo's last data freeze (3,514 samples there)

== Key References ==

If you are looking to learn about the project, I strongly recommend that you read the following papers

* Pilia G, Chen WM, Scuteri A, Orru M, Albai G, Dei M, Lai S, Usala G, Lai M, Loi P, Mameli C, Vacca L, Deiana M, Olla N, Masala M, Cao A, Najjar SS, Terracciano A, Nedorezov T, Sharov A, Zonderman AB, Abecasis GR, Costa P, Lakatta E and Schlessinger D (2006). Heritability of Cardiovascular and Personality Traits in 6,148 Sardinians. PLoS Genet 2:1207-1223 [[http://www.sph.umich.edu/csg/abecasis/publications/16934002.html Abstract and PDF]]

* Scuteri A, Sanna S, Chen WM, Uda M, Albai G, Strait J, Najjar S, Nagarajah R, Orru M, Usala G, Dei M, Lai S, Maschio A, Busonero F, Mulas A, Ehret GB, Fink AA, Weder A, Cooper R, Galan P, Chakravarti A, Schlessinger D, Cao A, Lakatta E and Abecasis GR (2007). Genome Wide Association Scan shows Genetic Variants in the FTO gene are Associated with Obesity Related Traits PLoS Genetics 3:1200-10 [[http://www.sph.umich.edu/csg/abecasis/publications/PLOS-Obesity-Scan.html Abstract and PDF]]

* Chen WM and Abecasis GR (2007). Family-based association tests for genomewide association scans. Am J Hum Genet 81:913-26 [[http://www.sph.umich.edu/csg/abecasis/publications/17924335.html Abstract and PDF]]

SardiNIA

2016-07-06T16:28:47Z

Kleckner: /* Future Directions */

== Project Leaders ==

* David Schlessinger (National Institutes on Aging, Baltimore)
* Manuela Uda (National Research Center, Cagliari, Italy)
* Goncalo Abecasis (University of Michigan, Ann Arbor)

== Genomewide Association Study ==

We carried out an initial genomewide association by genotyping 1405 samples with Affymetrix 500K SNP arrays. Then, because SardiNIA samples are closely related to each other, we were able to use these genotypes to impute the genomes of many close relatives who were genotyped with Affymetrix 10K SNP arrays. Typically, our SardiNIA GWAS analysis thus include approximately 4300 genotyped or imputed individuals. To learn more about the approach see Chen and Abecasis (2007) and Scuteri et al (2007).

=== Planned Updates ===

Several ongoing experiments are expected to gradually improve our GWAS data. First, we expect to integrate Affymetrix 1M SNP chip genotypes into our analyses. Second, we plan to genotype all sampled individuals with the Metabochip, which includes 200,000 SNPs. This will enable us to fill in genotypes for relatives of those genotyped with denser arrays more accurately, because we will more precisely identify shared stretches of chromosome.

== Medical Sequencing Project ==

We are sequencing the genomes of 1,000 individuals to learn about the genetics of blood lipid levels and personality.

== Status as of July 2016 ==

=== Locations of Files for Current Data Freeze of 3839 Samples===

NOTICE: We identified late in the process that two of the samples (22855 and 22385) were actually the same individual. They both should be the same individual 22855. Therefore, there are 3840 sample IDs in each of the files below, but only 22855 should move on to later processes. In future data freezes with this data, these two sequencing sets should be merged into a single 22855 individual.

* List of '''Sample Numbers'''
** The following file contains three columns: [SampleID used in these analyses] [ID supplied by CSCT or Sardinia or other project] [Sequencing core ID (if different)]:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/sampleIDConversion.txt

* List of paths to '''BAMs''' used in this data freeze (Index file)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_index_20150504.index
** 401 of these samples have some new BAM contribution since the previous data freeze... their BAMs can be found here: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/newsamples.index

* '''Pedigree''' (Not too helpful -- used for SNPCall)
** All Samples: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_ped_20150510.ped
** Disjoint Trios: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/Pedigree_Fall15_DataFreeze_Triplets.ped

* '''QC''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCStats.txt
** For a list of paths to all of the separate QPLOT files for each sample, see the file: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCFileListFinal.txt

* '''SNPCall''' Results (Produced with Gotcloud SNPCall and phased using Beagle4)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''IndelCall''' Results (Produced with Gotcloud Indel)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/indel/final/all.genotypes.sites.vcf.gz
** SNPEff and VEP declarations of Indel types can be found:
*** SNPEff: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/snpEff/*
*** VEP: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/VEP/*
** We used an '''Indel filtering strategy''' composed of many levels.
**# AC must be 1 or greater -- eliminate Indels with AC=0
**# the Indel should overlap with a VNTR region or overlap with another Indel. We used Adrian's annotate indels program to identify such overlaps. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/All.annotated.sites.vcf.gz
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/Overlaps.txt
**# at least 50% of the Indels should have informative AD field (we define "informative" to mean that the sample has U/(R+A+U)<0.50). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter2_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# at least 50% of the Indels should have informative PL field (we define "informative" to mean that the PL field for the sample is anything BUT ././. or 0/0/0). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter1_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# the Indel needs to have BF_LRE_LUD (a Bayes factor comparing a. related & HWE to b. unrelated & HWD) > -10. Higher BF_LRE_LUD should indicate a better Indel. We used Hyun's MiLK program to obtain BF_LRE_LUD values. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/all.genotypes.milk.sites.vcf
*** Overall results from all of the below filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/FinalIndelFilteringStatistics.txt
*** VCFs of Indels after filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.*.PASS.vcf.gz
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.PASS.vcf.gz

* Merged Indel VCF with beagles SNP VCF, then sorted to get the following VCFS:
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/chr*.PASS.beagled.vcf.gz

* Indel and SNP VCFs that have been combined AND BEAGLED AGAIN TOGETHER using Beagle4. '''These are the latest VCFS''':
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/beagle4/beagle4_chr*/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''mtDNA Copy Number''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/sardiniaCopyNumber_include.txt
** By sample copy number results by sample can be found: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/*.CopyNumber.noRand.500000.*.txt

* '''Filtering Samples''': Discrepancy Figures
** We compared chip data from previous work to the sequencing data produced now in hopes of identifying which samples have reliable sequencing data. This comparison was done for each chromosome separately and then combined into an overall discrepancy figure. Only 3189 of the 3840 samples (3188 of 3839 if you throw out sample with two sets of data) had chip data.
** By chromosome discrepancy figures for each sample can be found:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/*.diff.discordance_matrix
** By chromosome discrepancy figures for all samples in one file can be found:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/all.discordance.summary.txt

=== What is Complete ===
* SNP Call
** 24,901,469 SNPs passed filters
*** 16,822,922 are in dbSNP (67.6%)
*** %Known Ts/Tv - 2.24
*** %Novel Ts/Tv - 1.95
* InDel Call
** 1,194,945 passed filters

=== Future Directions ===
* NOTE: For sample filtering below, you will need to finish chromosome 1 for me. I have it currently running. Once it is done in a few days, you will need to run the command 'python calculateConcordance_onefile.py' while in the '/net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr1/' directory.
* '''Sample Filtering'''
** We did not do any filtering of samples (based on dupRate, genome coverage, mapping rate, proper paired, mean depth, or any other QPLOT stats) prior to SNP and Indel calling. Because of this, we want to do this filtering now. 3,188 or 3,839 samples have genome chip data from a few years ago. For these, we could look at the non-reference concordance between the chip genotypes and the sequencing genotypes and declare 'bad' samples to be those that fall below a certain threshold, such as 98% non-ref concordance. However, since the remaining 651 samples do not have chip data, this is not an option for them. Therefore, we decided on the following strategy instead:
**# Calculate non-reference concordance for the 3,188 samples that have chip data.
**# Create a prediction model using QPLOT statistics as predictors of non-reference concordance. Either do so on all of the 3,188 samples and look at R^2 (likely inflated from overfitting) or use cross-validation (test and training set) to give a measure of external predictive power.
**# If reasonable predictive power/R^2, use the prediction model to estimate the non-reference concordance amongst the 651 samples that do not have chip data. Also use the prediction model to estimate the non-reference concordance among the 3,188 samples that do have chip data.
**# Set a cut-off for 'good' versus 'bad' samples based on the estimated non-reference concordance and use it to filter samples.
** NOTE: The number of positions for which the chip data give 0/0 and sequencing gives 0/0, chip data gives 0/0 and sequencing gives 0/1, chip data gives 0/0 and sequencing gives 1/1, chip gives 0/1 and sequencing gives 0/0, etc. etc. BY chromosome can be found in the files /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/all.discordance.summary.txt. These can be used to calculate overall non-reference concordance across all chromosomes.
* '''Mitochondrial Depth Analysis'''
* '''Telomere Length Analysis'''
** Investigate the associations between telomere length (an indicator of aging) and variants. Likely interesting in Sardinia population because Sardinians have longer lifespans & centenarians.
* '''Phenotype Study'''
** Likely will not yield much because not many additional samples since Carlo's last data freeze (3,514 samples there)

== Key References ==

If you are looking to learn about the project, I strongly recommend that you read the following papers

* Pilia G, Chen WM, Scuteri A, Orru M, Albai G, Dei M, Lai S, Usala G, Lai M, Loi P, Mameli C, Vacca L, Deiana M, Olla N, Masala M, Cao A, Najjar SS, Terracciano A, Nedorezov T, Sharov A, Zonderman AB, Abecasis GR, Costa P, Lakatta E and Schlessinger D (2006). Heritability of Cardiovascular and Personality Traits in 6,148 Sardinians. PLoS Genet 2:1207-1223 [[http://www.sph.umich.edu/csg/abecasis/publications/16934002.html Abstract and PDF]]

* Scuteri A, Sanna S, Chen WM, Uda M, Albai G, Strait J, Najjar S, Nagarajah R, Orru M, Usala G, Dei M, Lai S, Maschio A, Busonero F, Mulas A, Ehret GB, Fink AA, Weder A, Cooper R, Galan P, Chakravarti A, Schlessinger D, Cao A, Lakatta E and Abecasis GR (2007). Genome Wide Association Scan shows Genetic Variants in the FTO gene are Associated with Obesity Related Traits PLoS Genetics 3:1200-10 [[http://www.sph.umich.edu/csg/abecasis/publications/PLOS-Obesity-Scan.html Abstract and PDF]]

* Chen WM and Abecasis GR (2007). Family-based association tests for genomewide association scans. Am J Hum Genet 81:913-26 [[http://www.sph.umich.edu/csg/abecasis/publications/17924335.html Abstract and PDF]]

SardiNIA

2016-07-06T16:27:27Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

== Project Leaders ==

* David Schlessinger (National Institutes on Aging, Baltimore)
* Manuela Uda (National Research Center, Cagliari, Italy)
* Goncalo Abecasis (University of Michigan, Ann Arbor)

== Genomewide Association Study ==

We carried out an initial genomewide association by genotyping 1405 samples with Affymetrix 500K SNP arrays. Then, because SardiNIA samples are closely related to each other, we were able to use these genotypes to impute the genomes of many close relatives who were genotyped with Affymetrix 10K SNP arrays. Typically, our SardiNIA GWAS analysis thus include approximately 4300 genotyped or imputed individuals. To learn more about the approach see Chen and Abecasis (2007) and Scuteri et al (2007).

=== Planned Updates ===

Several ongoing experiments are expected to gradually improve our GWAS data. First, we expect to integrate Affymetrix 1M SNP chip genotypes into our analyses. Second, we plan to genotype all sampled individuals with the Metabochip, which includes 200,000 SNPs. This will enable us to fill in genotypes for relatives of those genotyped with denser arrays more accurately, because we will more precisely identify shared stretches of chromosome.

== Medical Sequencing Project ==

We are sequencing the genomes of 1,000 individuals to learn about the genetics of blood lipid levels and personality.

== Status as of July 2016 ==

=== Locations of Files for Current Data Freeze of 3839 Samples===

NOTICE: We identified late in the process that two of the samples (22855 and 22385) were actually the same individual. They both should be the same individual 22855. Therefore, there are 3840 sample IDs in each of the files below, but only 22855 should move on to later processes. In future data freezes with this data, these two sequencing sets should be merged into a single 22855 individual.

* List of '''Sample Numbers'''
** The following file contains three columns: [SampleID used in these analyses] [ID supplied by CSCT or Sardinia or other project] [Sequencing core ID (if different)]:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/sampleIDConversion.txt

* List of paths to '''BAMs''' used in this data freeze (Index file)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_index_20150504.index
** 401 of these samples have some new BAM contribution since the previous data freeze... their BAMs can be found here: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/newsamples.index

* '''Pedigree''' (Not too helpful -- used for SNPCall)
** All Samples: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_ped_20150510.ped
** Disjoint Trios: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/Pedigree_Fall15_DataFreeze_Triplets.ped

* '''QC''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCStats.txt
** For a list of paths to all of the separate QPLOT files for each sample, see the file: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCFileListFinal.txt

* '''SNPCall''' Results (Produced with Gotcloud SNPCall and phased using Beagle4)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''IndelCall''' Results (Produced with Gotcloud Indel)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/indel/final/all.genotypes.sites.vcf.gz
** SNPEff and VEP declarations of Indel types can be found:
*** SNPEff: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/snpEff/*
*** VEP: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/VEP/*
** We used an '''Indel filtering strategy''' composed of many levels.
**# AC must be 1 or greater -- eliminate Indels with AC=0
**# the Indel should overlap with a VNTR region or overlap with another Indel. We used Adrian's annotate indels program to identify such overlaps. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/All.annotated.sites.vcf.gz
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/Overlaps.txt
**# at least 50% of the Indels should have informative AD field (we define "informative" to mean that the sample has U/(R+A+U)<0.50). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter2_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# at least 50% of the Indels should have informative PL field (we define "informative" to mean that the PL field for the sample is anything BUT ././. or 0/0/0). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter1_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# the Indel needs to have BF_LRE_LUD (a Bayes factor comparing a. related & HWE to b. unrelated & HWD) > -10. Higher BF_LRE_LUD should indicate a better Indel. We used Hyun's MiLK program to obtain BF_LRE_LUD values. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/all.genotypes.milk.sites.vcf
*** Overall results from all of the below filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/FinalIndelFilteringStatistics.txt
*** VCFs of Indels after filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.*.PASS.vcf.gz
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.PASS.vcf.gz

* Merged Indel VCF with beagles SNP VCF, then sorted to get the following VCFS:
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/chr*.PASS.beagled.vcf.gz

* Indel and SNP VCFs that have been combined AND BEAGLED AGAIN TOGETHER using Beagle4. '''These are the latest VCFS''':
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/beagle4/beagle4_chr*/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''mtDNA Copy Number''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/sardiniaCopyNumber_include.txt
** By sample copy number results by sample can be found: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/*.CopyNumber.noRand.500000.*.txt

* '''Filtering Samples''': Discrepancy Figures
** We compared chip data from previous work to the sequencing data produced now in hopes of identifying which samples have reliable sequencing data. This comparison was done for each chromosome separately and then combined into an overall discrepancy figure. Only 3189 of the 3840 samples (3188 of 3839 if you throw out sample with two sets of data) had chip data.
** By chromosome discrepancy figures for each sample can be found:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/*.diff.discordance_matrix
** By chromosome discrepancy figures for all samples in one file can be found:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/all.discordance.summary.txt

=== What is Complete ===
* SNP Call
** 24,901,469 SNPs passed filters
*** 16,822,922 are in dbSNP (67.6%)
*** %Known Ts/Tv - 2.24
*** %Novel Ts/Tv - 1.95
* InDel Call
** 1,194,945 passed filters

=== Future Directions ===
* '''Sample Filtering'''
** We did not do any filtering of samples (based on dupRate, genome coverage, mapping rate, proper paired, mean depth, or any other QPLOT stats) prior to SNP and Indel calling. Because of this, we want to do this filtering now. 3,188 or 3,839 samples have genome chip data from a few years ago. For these, we could look at the non-reference concordance between the chip genotypes and the sequencing genotypes and declare 'bad' samples to be those that fall below a certain threshold, such as 98% non-ref concordance. However, since the remaining 651 samples do not have chip data, this is not an option for them. Therefore, we decided on the following strategy instead:
**# Calculate non-reference concordance for the 3,188 samples that have chip data.
**# Create a prediction model using QPLOT statistics as predictors of non-reference concordance. Either do so on all of the 3,188 samples and look at R^2 (likely inflated from overfitting) or use cross-validation (test and training set) to give a measure of external predictive power.
**# If reasonable predictive power/R^2, use the prediction model to estimate the non-reference concordance amongst the 651 samples that do not have chip data. Also use the prediction model to estimate the non-reference concordance among the 3,188 samples that do have chip data.
**# Set a cut-off for 'good' versus 'bad' samples based on the estimated non-reference concordance and use it to filter samples.
** NOTE: The number of positions for which the chip data give 0/0 and sequencing gives 0/0, chip data gives 0/0 and sequencing gives 0/1, chip data gives 0/0 and sequencing gives 1/1, chip gives 0/1 and sequencing gives 0/0, etc. etc. BY chromosome can be found in the files /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/all.discordance.summary.txt. These can be used to calculate overall non-reference concordance across all chromosomes.
* '''Mitochondrial Depth Analysis'''
* '''Telomere Length Analysis'''
** Investigate the associations between telomere length (an indicator of aging) and variants. Likely interesting in Sardinia population because Sardinians have longer lifespans & centenarians.
* '''Phenotype Study'''
** Likely will not yield much because not many additional samples since Carlo's last data freeze (3,514 samples there)

== Key References ==

If you are looking to learn about the project, I strongly recommend that you read the following papers

* Pilia G, Chen WM, Scuteri A, Orru M, Albai G, Dei M, Lai S, Usala G, Lai M, Loi P, Mameli C, Vacca L, Deiana M, Olla N, Masala M, Cao A, Najjar SS, Terracciano A, Nedorezov T, Sharov A, Zonderman AB, Abecasis GR, Costa P, Lakatta E and Schlessinger D (2006). Heritability of Cardiovascular and Personality Traits in 6,148 Sardinians. PLoS Genet 2:1207-1223 [[http://www.sph.umich.edu/csg/abecasis/publications/16934002.html Abstract and PDF]]

* Scuteri A, Sanna S, Chen WM, Uda M, Albai G, Strait J, Najjar S, Nagarajah R, Orru M, Usala G, Dei M, Lai S, Maschio A, Busonero F, Mulas A, Ehret GB, Fink AA, Weder A, Cooper R, Galan P, Chakravarti A, Schlessinger D, Cao A, Lakatta E and Abecasis GR (2007). Genome Wide Association Scan shows Genetic Variants in the FTO gene are Associated with Obesity Related Traits PLoS Genetics 3:1200-10 [[http://www.sph.umich.edu/csg/abecasis/publications/PLOS-Obesity-Scan.html Abstract and PDF]]

* Chen WM and Abecasis GR (2007). Family-based association tests for genomewide association scans. Am J Hum Genet 81:913-26 [[http://www.sph.umich.edu/csg/abecasis/publications/17924335.html Abstract and PDF]]

SardiNIA

2016-07-06T16:27:02Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

== Project Leaders ==

* David Schlessinger (National Institutes on Aging, Baltimore)
* Manuela Uda (National Research Center, Cagliari, Italy)
* Goncalo Abecasis (University of Michigan, Ann Arbor)

== Genomewide Association Study ==

We carried out an initial genomewide association by genotyping 1405 samples with Affymetrix 500K SNP arrays. Then, because SardiNIA samples are closely related to each other, we were able to use these genotypes to impute the genomes of many close relatives who were genotyped with Affymetrix 10K SNP arrays. Typically, our SardiNIA GWAS analysis thus include approximately 4300 genotyped or imputed individuals. To learn more about the approach see Chen and Abecasis (2007) and Scuteri et al (2007).

=== Planned Updates ===

Several ongoing experiments are expected to gradually improve our GWAS data. First, we expect to integrate Affymetrix 1M SNP chip genotypes into our analyses. Second, we plan to genotype all sampled individuals with the Metabochip, which includes 200,000 SNPs. This will enable us to fill in genotypes for relatives of those genotyped with denser arrays more accurately, because we will more precisely identify shared stretches of chromosome.

== Medical Sequencing Project ==

We are sequencing the genomes of 1,000 individuals to learn about the genetics of blood lipid levels and personality.

== Status as of July 2016 ==

=== Locations of Files for Current Data Freeze of 3839 Samples===

NOTICE: We identified late in the process that two of the samples (22855 and 22385) were actually the same individual. They both should be the same individual 22855. Therefore, there are 3840 sample IDs in each of the files below, but only 22855 should move on to later processes. In future data freezes with this data, these two sequencing sets should be merged into a single 22855 individual.

* List of '''Sample Numbers'''
** The following file contains three columns: [SampleID used in these analyses] [ID supplied by CSCT or Sardinia or other project] [Sequencing core ID (if different)]:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/sampleIDConversion.txt

* List of paths to '''BAMs''' used in this data freeze (Index file)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_index_20150504.index
** 401 of these samples have some new BAM contribution since the previous data freeze... their BAMs can be found here: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/newsamples.index

* '''Pedigree''' (Not too helpful -- used for SNPCall)
** All Samples: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_ped_20150510.ped
** Disjoint Trios: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/Pedigree_Fall15_DataFreeze_Triplets.ped

* '''QC''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCStats.txt
** For a list of paths to all of the separate QPLOT files for each sample, see the file: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCFileListFinal.txt

* '''SNPCall''' Results (Produced with Gotcloud SNPCall and phased using Beagle4)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''IndelCall''' Results (Produced with Gotcloud Indel)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/indel/final/all.genotypes.sites.vcf.gz
** SNPEff and VEP declarations of Indel types can be found:
*** SNPEff: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/snpEff/*
*** VEP: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/VEP/*
** We used an '''Indel filtering strategy''' composed of many levels.
**# AC must be 1 or greater -- eliminate Indels with AC=0
**# the Indel should overlap with a VNTR region or overlap with another Indel. We used Adrian's annotate indels program to identify such overlaps. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/All.annotated.sites.vcf.gz
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/Overlaps.txt
**# at least 50% of the Indels should have informative AD field (we define "informative" to mean that the sample has U/(R+A+U)<0.50). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter2_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# at least 50% of the Indels should have informative PL field (we define "informative" to mean that the PL field for the sample is anything BUT ././. or 0/0/0). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter1_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# the Indel needs to have BF_LRE_LUD (a Bayes factor comparing a. related & HWE to b. unrelated & HWD) > -10. Higher BF_LRE_LUD should indicate a better Indel. We used Hyun's MiLK program to obtain BF_LRE_LUD values. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/all.genotypes.milk.sites.vcf
*** Overall results from all of the below filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/FinalIndelFilteringStatistics.txt
*** VCFs of Indels after filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.*.PASS.vcf.gz
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.PASS.vcf.gz

* Merged Indel VCF with beagles SNP VCF, then sorted to get the following VCFS:
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/chr*.PASS.beagled.vcf.gz

* Indel and SNP VCFs that have been combined AND BEAGLED AGAIN TOGETHER using Beagle4. '''These are the latest VCFS''':
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/beagle4/beagle4_chr*/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''mtDNA Copy Number''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/sardiniaCopyNumber_include.txt
** By sample copy number results by sample can be found: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/*.CopyNumber.noRand.500000.*.txt

* '''Filtering Samples''': Discrepancy Figures
** We compared chip data from previous work to the sequencing data produced now in hopes of identifying which samples have reliable sequencing data. This comparison was done for each chromosome separately and then combined into an overall discrepancy figure. Only 3189 of the 3840 samples (3188 of 3839 if you throw out sample with two sets of data) had chip data.
** By chromosome discrepancy figures for each sample can be found:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/*.diff.discordance_matrix
** By chromosome discrepancy figures for all samples in ones file can be found:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/all.discordance.summary.txt

=== What is Complete ===
* SNP Call
** 24,901,469 SNPs passed filters
*** 16,822,922 are in dbSNP (67.6%)
*** %Known Ts/Tv - 2.24
*** %Novel Ts/Tv - 1.95
* InDel Call
** 1,194,945 passed filters

=== Future Directions ===
* '''Sample Filtering'''
** We did not do any filtering of samples (based on dupRate, genome coverage, mapping rate, proper paired, mean depth, or any other QPLOT stats) prior to SNP and Indel calling. Because of this, we want to do this filtering now. 3,188 or 3,839 samples have genome chip data from a few years ago. For these, we could look at the non-reference concordance between the chip genotypes and the sequencing genotypes and declare 'bad' samples to be those that fall below a certain threshold, such as 98% non-ref concordance. However, since the remaining 651 samples do not have chip data, this is not an option for them. Therefore, we decided on the following strategy instead:
**# Calculate non-reference concordance for the 3,188 samples that have chip data.
**# Create a prediction model using QPLOT statistics as predictors of non-reference concordance. Either do so on all of the 3,188 samples and look at R^2 (likely inflated from overfitting) or use cross-validation (test and training set) to give a measure of external predictive power.
**# If reasonable predictive power/R^2, use the prediction model to estimate the non-reference concordance amongst the 651 samples that do not have chip data. Also use the prediction model to estimate the non-reference concordance among the 3,188 samples that do have chip data.
**# Set a cut-off for 'good' versus 'bad' samples based on the estimated non-reference concordance and use it to filter samples.
** NOTE: The number of positions for which the chip data give 0/0 and sequencing gives 0/0, chip data gives 0/0 and sequencing gives 0/1, chip data gives 0/0 and sequencing gives 1/1, chip gives 0/1 and sequencing gives 0/0, etc. etc. BY chromosome can be found in the files /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/all.discordance.summary.txt. These can be used to calculate overall non-reference concordance across all chromosomes.
* '''Mitochondrial Depth Analysis'''
* '''Telomere Length Analysis'''
** Investigate the associations between telomere length (an indicator of aging) and variants. Likely interesting in Sardinia population because Sardinians have longer lifespans & centenarians.
* '''Phenotype Study'''
** Likely will not yield much because not many additional samples since Carlo's last data freeze (3,514 samples there)

== Key References ==

If you are looking to learn about the project, I strongly recommend that you read the following papers

* Pilia G, Chen WM, Scuteri A, Orru M, Albai G, Dei M, Lai S, Usala G, Lai M, Loi P, Mameli C, Vacca L, Deiana M, Olla N, Masala M, Cao A, Najjar SS, Terracciano A, Nedorezov T, Sharov A, Zonderman AB, Abecasis GR, Costa P, Lakatta E and Schlessinger D (2006). Heritability of Cardiovascular and Personality Traits in 6,148 Sardinians. PLoS Genet 2:1207-1223 [[http://www.sph.umich.edu/csg/abecasis/publications/16934002.html Abstract and PDF]]

* Scuteri A, Sanna S, Chen WM, Uda M, Albai G, Strait J, Najjar S, Nagarajah R, Orru M, Usala G, Dei M, Lai S, Maschio A, Busonero F, Mulas A, Ehret GB, Fink AA, Weder A, Cooper R, Galan P, Chakravarti A, Schlessinger D, Cao A, Lakatta E and Abecasis GR (2007). Genome Wide Association Scan shows Genetic Variants in the FTO gene are Associated with Obesity Related Traits PLoS Genetics 3:1200-10 [[http://www.sph.umich.edu/csg/abecasis/publications/PLOS-Obesity-Scan.html Abstract and PDF]]

* Chen WM and Abecasis GR (2007). Family-based association tests for genomewide association scans. Am J Hum Genet 81:913-26 [[http://www.sph.umich.edu/csg/abecasis/publications/17924335.html Abstract and PDF]]

SardiNIA

2016-07-06T16:25:58Z

Kleckner: /* Future Directions */

== Project Leaders ==

* David Schlessinger (National Institutes on Aging, Baltimore)
* Manuela Uda (National Research Center, Cagliari, Italy)
* Goncalo Abecasis (University of Michigan, Ann Arbor)

== Genomewide Association Study ==

We carried out an initial genomewide association by genotyping 1405 samples with Affymetrix 500K SNP arrays. Then, because SardiNIA samples are closely related to each other, we were able to use these genotypes to impute the genomes of many close relatives who were genotyped with Affymetrix 10K SNP arrays. Typically, our SardiNIA GWAS analysis thus include approximately 4300 genotyped or imputed individuals. To learn more about the approach see Chen and Abecasis (2007) and Scuteri et al (2007).

=== Planned Updates ===

Several ongoing experiments are expected to gradually improve our GWAS data. First, we expect to integrate Affymetrix 1M SNP chip genotypes into our analyses. Second, we plan to genotype all sampled individuals with the Metabochip, which includes 200,000 SNPs. This will enable us to fill in genotypes for relatives of those genotyped with denser arrays more accurately, because we will more precisely identify shared stretches of chromosome.

== Medical Sequencing Project ==

We are sequencing the genomes of 1,000 individuals to learn about the genetics of blood lipid levels and personality.

== Status as of July 2016 ==

=== Locations of Files for Current Data Freeze of 3839 Samples===

NOTICE: We identified late in the process that two of the samples (22855 and 22385) were actually the same individual. They both should be the same individual 22855. Therefore, there are 3840 sample IDs in each of the files below, but only 22855 should move on to later processes. In future data freezes with this data, these two sequencing sets should be merged into a single 22855 individual.

* List of '''Sample Numbers'''
** The following file contains three columns: [SampleID used in these analyses] [ID supplied by CSCT or Sardinia or other project] [Sequencing core ID (if different)]:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/sampleIDConversion.txt

* List of paths to '''BAMs''' used in this data freeze (Index file)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_index_20150504.index
** 401 of these samples have some new BAM contribution since the previous data freeze... their BAMs can be found here: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/newsamples.index

* '''Pedigree''' (Not too helpful -- used for SNPCall)
** All Samples: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_ped_20150510.ped
** Disjoint Trios: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/Pedigree_Fall15_DataFreeze_Triplets.ped

* '''QC''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCStats.txt
** For a list of paths to all of the separate QPLOT files for each sample, see the file: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCFileListFinal.txt

* '''SNPCall''' Results (Produced with Gotcloud SNPCall and phased using Beagle4)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''IndelCall''' Results (Produced with Gotcloud Indel)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/indel/final/all.genotypes.sites.vcf.gz
** SNPEff and VEP declarations of Indel types can be found:
*** SNPEff: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/snpEff/*
*** VEP: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/VEP/*
** We used an '''Indel filtering strategy''' composed of many levels.
**# AC must be 1 or greater -- eliminate Indels with AC=0
**# the Indel should overlap with a VNTR region or overlap with another Indel. We used Adrian's annotate indels program to identify such overlaps. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/All.annotated.sites.vcf.gz
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/Overlaps.txt
**# at least 50% of the Indels should have informative AD field (we define "informative" to mean that the sample has U/(R+A+U)<0.50). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter2_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# at least 50% of the Indels should have informative PL field (we define "informative" to mean that the PL field for the sample is anything BUT ././. or 0/0/0). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter1_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# the Indel needs to have BF_LRE_LUD (a Bayes factor comparing a. related & HWE to b. unrelated & HWD) > -10. Higher BF_LRE_LUD should indicate a better Indel. We used Hyun's MiLK program to obtain BF_LRE_LUD values. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/all.genotypes.milk.sites.vcf
*** Overall results from all of the below filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/FinalIndelFilteringStatistics.txt
*** VCFs of Indels after filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.*.PASS.vcf.gz
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.PASS.vcf.gz

* Merged Indel VCF with beagles SNP VCF, then sorted to get the following VCFS:
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/chr*.PASS.beagled.vcf.gz

* Indel and SNP VCFs that have been combined AND BEAGLED AGAIN TOGETHER using Beagle4. '''These are the latest VCFS''':
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/beagle4/beagle4_chr*/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''mtDNA Copy Number''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/sardiniaCopyNumber_include.txt
** By sample copy number results by sample can be found: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/*.CopyNumber.noRand.500000.*.txt

* '''Filtering Samples''': Discrepancy Figures
** We compared chip data from previous work to the sequencing data produced now in hopes of identifying which samples have reliable sequencing data. This comparison was done for each chromosome separately and then combined into an overall discrepancy figure. Only 3189 of the 3840 samples (3188 of 3839 if you throw out sample with two sets of data) had
** By chromosome discrepancy figures can be found:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/*.diff.discordance_matrix

=== What is Complete ===
* SNP Call
** 24,901,469 SNPs passed filters
*** 16,822,922 are in dbSNP (67.6%)
*** %Known Ts/Tv - 2.24
*** %Novel Ts/Tv - 1.95
* InDel Call
** 1,194,945 passed filters

=== Future Directions ===
* '''Sample Filtering'''
** We did not do any filtering of samples (based on dupRate, genome coverage, mapping rate, proper paired, mean depth, or any other QPLOT stats) prior to SNP and Indel calling. Because of this, we want to do this filtering now. 3,188 or 3,839 samples have genome chip data from a few years ago. For these, we could look at the non-reference concordance between the chip genotypes and the sequencing genotypes and declare 'bad' samples to be those that fall below a certain threshold, such as 98% non-ref concordance. However, since the remaining 651 samples do not have chip data, this is not an option for them. Therefore, we decided on the following strategy instead:
**# Calculate non-reference concordance for the 3,188 samples that have chip data.
**# Create a prediction model using QPLOT statistics as predictors of non-reference concordance. Either do so on all of the 3,188 samples and look at R^2 (likely inflated from overfitting) or use cross-validation (test and training set) to give a measure of external predictive power.
**# If reasonable predictive power/R^2, use the prediction model to estimate the non-reference concordance amongst the 651 samples that do not have chip data. Also use the prediction model to estimate the non-reference concordance among the 3,188 samples that do have chip data.
**# Set a cut-off for 'good' versus 'bad' samples based on the estimated non-reference concordance and use it to filter samples.
** NOTE: The number of positions for which the chip data give 0/0 and sequencing gives 0/0, chip data gives 0/0 and sequencing gives 0/1, chip data gives 0/0 and sequencing gives 1/1, chip gives 0/1 and sequencing gives 0/0, etc. etc. BY chromosome can be found in the files /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/all.discordance.summary.txt. These can be used to calculate overall non-reference concordance across all chromosomes.
* '''Mitochondrial Depth Analysis'''
* '''Telomere Length Analysis'''
** Investigate the associations between telomere length (an indicator of aging) and variants. Likely interesting in Sardinia population because Sardinians have longer lifespans & centenarians.
* '''Phenotype Study'''
** Likely will not yield much because not many additional samples since Carlo's last data freeze (3,514 samples there)

== Key References ==

If you are looking to learn about the project, I strongly recommend that you read the following papers

* Pilia G, Chen WM, Scuteri A, Orru M, Albai G, Dei M, Lai S, Usala G, Lai M, Loi P, Mameli C, Vacca L, Deiana M, Olla N, Masala M, Cao A, Najjar SS, Terracciano A, Nedorezov T, Sharov A, Zonderman AB, Abecasis GR, Costa P, Lakatta E and Schlessinger D (2006). Heritability of Cardiovascular and Personality Traits in 6,148 Sardinians. PLoS Genet 2:1207-1223 [[http://www.sph.umich.edu/csg/abecasis/publications/16934002.html Abstract and PDF]]

* Scuteri A, Sanna S, Chen WM, Uda M, Albai G, Strait J, Najjar S, Nagarajah R, Orru M, Usala G, Dei M, Lai S, Maschio A, Busonero F, Mulas A, Ehret GB, Fink AA, Weder A, Cooper R, Galan P, Chakravarti A, Schlessinger D, Cao A, Lakatta E and Abecasis GR (2007). Genome Wide Association Scan shows Genetic Variants in the FTO gene are Associated with Obesity Related Traits PLoS Genetics 3:1200-10 [[http://www.sph.umich.edu/csg/abecasis/publications/PLOS-Obesity-Scan.html Abstract and PDF]]

* Chen WM and Abecasis GR (2007). Family-based association tests for genomewide association scans. Am J Hum Genet 81:913-26 [[http://www.sph.umich.edu/csg/abecasis/publications/17924335.html Abstract and PDF]]

SardiNIA

2016-07-06T15:58:36Z

Kleckner: /* Future Directions */

== Project Leaders ==

* David Schlessinger (National Institutes on Aging, Baltimore)
* Manuela Uda (National Research Center, Cagliari, Italy)
* Goncalo Abecasis (University of Michigan, Ann Arbor)

== Genomewide Association Study ==

We carried out an initial genomewide association by genotyping 1405 samples with Affymetrix 500K SNP arrays. Then, because SardiNIA samples are closely related to each other, we were able to use these genotypes to impute the genomes of many close relatives who were genotyped with Affymetrix 10K SNP arrays. Typically, our SardiNIA GWAS analysis thus include approximately 4300 genotyped or imputed individuals. To learn more about the approach see Chen and Abecasis (2007) and Scuteri et al (2007).

=== Planned Updates ===

Several ongoing experiments are expected to gradually improve our GWAS data. First, we expect to integrate Affymetrix 1M SNP chip genotypes into our analyses. Second, we plan to genotype all sampled individuals with the Metabochip, which includes 200,000 SNPs. This will enable us to fill in genotypes for relatives of those genotyped with denser arrays more accurately, because we will more precisely identify shared stretches of chromosome.

== Medical Sequencing Project ==

We are sequencing the genomes of 1,000 individuals to learn about the genetics of blood lipid levels and personality.

== Status as of July 2016 ==

=== Locations of Files for Current Data Freeze of 3839 Samples===

NOTICE: We identified late in the process that two of the samples (22855 and 22385) were actually the same individual. They both should be the same individual 22855. Therefore, there are 3840 sample IDs in each of the files below, but only 22855 should move on to later processes. In future data freezes with this data, these two sequencing sets should be merged into a single 22855 individual.

* List of '''Sample Numbers'''
** The following file contains three columns: [SampleID used in these analyses] [ID supplied by CSCT or Sardinia or other project] [Sequencing core ID (if different)]:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/sampleIDConversion.txt

* List of paths to '''BAMs''' used in this data freeze (Index file)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_index_20150504.index
** 401 of these samples have some new BAM contribution since the previous data freeze... their BAMs can be found here: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/newsamples.index

* '''Pedigree''' (Not too helpful -- used for SNPCall)
** All Samples: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_ped_20150510.ped
** Disjoint Trios: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/Pedigree_Fall15_DataFreeze_Triplets.ped

* '''QC''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCStats.txt
** For a list of paths to all of the separate QPLOT files for each sample, see the file: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCFileListFinal.txt

* '''SNPCall''' Results (Produced with Gotcloud SNPCall and phased using Beagle4)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''IndelCall''' Results (Produced with Gotcloud Indel)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/indel/final/all.genotypes.sites.vcf.gz
** SNPEff and VEP declarations of Indel types can be found:
*** SNPEff: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/snpEff/*
*** VEP: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/VEP/*
** We used an '''Indel filtering strategy''' composed of many levels.
**# AC must be 1 or greater -- eliminate Indels with AC=0
**# the Indel should overlap with a VNTR region or overlap with another Indel. We used Adrian's annotate indels program to identify such overlaps. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/All.annotated.sites.vcf.gz
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/Overlaps.txt
**# at least 50% of the Indels should have informative AD field (we define "informative" to mean that the sample has U/(R+A+U)<0.50). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter2_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# at least 50% of the Indels should have informative PL field (we define "informative" to mean that the PL field for the sample is anything BUT ././. or 0/0/0). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter1_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# the Indel needs to have BF_LRE_LUD (a Bayes factor comparing a. related & HWE to b. unrelated & HWD) > -10. Higher BF_LRE_LUD should indicate a better Indel. We used Hyun's MiLK program to obtain BF_LRE_LUD values. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/all.genotypes.milk.sites.vcf
*** Overall results from all of the below filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/FinalIndelFilteringStatistics.txt
*** VCFs of Indels after filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.*.PASS.vcf.gz
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.PASS.vcf.gz

* Merged Indel VCF with beagles SNP VCF, then sorted to get the following VCFS:
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/chr*.PASS.beagled.vcf.gz

* Indel and SNP VCFs that have been combined AND BEAGLED AGAIN TOGETHER using Beagle4. '''These are the latest VCFS''':
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/beagle4/beagle4_chr*/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''mtDNA Copy Number''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/sardiniaCopyNumber_include.txt
** By sample copy number results by sample can be found: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/*.CopyNumber.noRand.500000.*.txt

* '''Filtering Samples''': Discrepancy Figures
** We compared chip data from previous work to the sequencing data produced now in hopes of identifying which samples have reliable sequencing data. This comparison was done for each chromosome separately and then combined into an overall discrepancy figure. Only 3189 of the 3840 samples (3188 of 3839 if you throw out sample with two sets of data) had
** By chromosome discrepancy figures can be found:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/*.diff.discordance_matrix

=== What is Complete ===
* SNP Call
** 24,901,469 SNPs passed filters
*** 16,822,922 are in dbSNP (67.6%)
*** %Known Ts/Tv - 2.24
*** %Novel Ts/Tv - 1.95
* InDel Call
** 1,194,945 passed filters

=== Future Directions ===
* '''Sample Filtering'''
** We did not do any filtering of samples (based on dupRate, genome coverage, mapping rate, proper paired, mean depth, or any other QPLOT stats) prior to SNP and Indel calling. Because of this, we want to do this filtering now. 3,188 or 3,839 samples have genome chip data from a few years ago. For these, we could look at the non-reference concordance between the chip genotypes and the sequencing genotypes and declare 'bad' samples to be those that fall below a certain threshold, such as 98% non-ref concordance. However, since the remaining 651 samples do not have chip data, this is not an option for them. Therefore, we decided on the following strategy instead:
**# Calculate non-reference concordance for the 3,188 samples that have chip data.
**# Create a prediction model using QPLOT statistics as predictors of non-reference concordance. Either do so on all of the 3,188 samples and look at R^2 (likely inflated from overfitting) or use cross-validation (test and training set) to give a measure of external predictive power.
**# If reasonable predictive power/R^2, use the prediction model to estimate the non-reference concordance amongst the 651 samples that do not have chip data. Also use the prediction model to estimate the non-reference concordance among the 3,188 samples that do have chip data.
**# Set a cut-off for 'good' versus 'bad' samples based on the estimated non-reference concordance and use it to filter samples.
* '''Mitochondrial Depth Analysis'''
* '''Telomere Length Analysis'''
** Investigate the associations between telomere length (an indicator of aging) and variants. Likely interesting in Sardinia population because Sardinians have longer lifespans & centenarians.
* '''Phenotype Study'''
** Likely will not yield much because not many additional samples since Carlo's last data freeze (3,514 samples there)

== Key References ==

If you are looking to learn about the project, I strongly recommend that you read the following papers

* Pilia G, Chen WM, Scuteri A, Orru M, Albai G, Dei M, Lai S, Usala G, Lai M, Loi P, Mameli C, Vacca L, Deiana M, Olla N, Masala M, Cao A, Najjar SS, Terracciano A, Nedorezov T, Sharov A, Zonderman AB, Abecasis GR, Costa P, Lakatta E and Schlessinger D (2006). Heritability of Cardiovascular and Personality Traits in 6,148 Sardinians. PLoS Genet 2:1207-1223 [[http://www.sph.umich.edu/csg/abecasis/publications/16934002.html Abstract and PDF]]

* Scuteri A, Sanna S, Chen WM, Uda M, Albai G, Strait J, Najjar S, Nagarajah R, Orru M, Usala G, Dei M, Lai S, Maschio A, Busonero F, Mulas A, Ehret GB, Fink AA, Weder A, Cooper R, Galan P, Chakravarti A, Schlessinger D, Cao A, Lakatta E and Abecasis GR (2007). Genome Wide Association Scan shows Genetic Variants in the FTO gene are Associated with Obesity Related Traits PLoS Genetics 3:1200-10 [[http://www.sph.umich.edu/csg/abecasis/publications/PLOS-Obesity-Scan.html Abstract and PDF]]

* Chen WM and Abecasis GR (2007). Family-based association tests for genomewide association scans. Am J Hum Genet 81:913-26 [[http://www.sph.umich.edu/csg/abecasis/publications/17924335.html Abstract and PDF]]

SardiNIA

2016-07-06T15:55:16Z

Kleckner: /* Future Directions */

== Project Leaders ==

* David Schlessinger (National Institutes on Aging, Baltimore)
* Manuela Uda (National Research Center, Cagliari, Italy)
* Goncalo Abecasis (University of Michigan, Ann Arbor)

== Genomewide Association Study ==

We carried out an initial genomewide association by genotyping 1405 samples with Affymetrix 500K SNP arrays. Then, because SardiNIA samples are closely related to each other, we were able to use these genotypes to impute the genomes of many close relatives who were genotyped with Affymetrix 10K SNP arrays. Typically, our SardiNIA GWAS analysis thus include approximately 4300 genotyped or imputed individuals. To learn more about the approach see Chen and Abecasis (2007) and Scuteri et al (2007).

=== Planned Updates ===

Several ongoing experiments are expected to gradually improve our GWAS data. First, we expect to integrate Affymetrix 1M SNP chip genotypes into our analyses. Second, we plan to genotype all sampled individuals with the Metabochip, which includes 200,000 SNPs. This will enable us to fill in genotypes for relatives of those genotyped with denser arrays more accurately, because we will more precisely identify shared stretches of chromosome.

== Medical Sequencing Project ==

We are sequencing the genomes of 1,000 individuals to learn about the genetics of blood lipid levels and personality.

== Status as of July 2016 ==

=== Locations of Files for Current Data Freeze of 3839 Samples===

NOTICE: We identified late in the process that two of the samples (22855 and 22385) were actually the same individual. They both should be the same individual 22855. Therefore, there are 3840 sample IDs in each of the files below, but only 22855 should move on to later processes. In future data freezes with this data, these two sequencing sets should be merged into a single 22855 individual.

* List of '''Sample Numbers'''
** The following file contains three columns: [SampleID used in these analyses] [ID supplied by CSCT or Sardinia or other project] [Sequencing core ID (if different)]:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/sampleIDConversion.txt

* List of paths to '''BAMs''' used in this data freeze (Index file)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_index_20150504.index
** 401 of these samples have some new BAM contribution since the previous data freeze... their BAMs can be found here: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/newsamples.index

* '''Pedigree''' (Not too helpful -- used for SNPCall)
** All Samples: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_ped_20150510.ped
** Disjoint Trios: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/Pedigree_Fall15_DataFreeze_Triplets.ped

* '''QC''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCStats.txt
** For a list of paths to all of the separate QPLOT files for each sample, see the file: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCFileListFinal.txt

* '''SNPCall''' Results (Produced with Gotcloud SNPCall and phased using Beagle4)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''IndelCall''' Results (Produced with Gotcloud Indel)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/indel/final/all.genotypes.sites.vcf.gz
** SNPEff and VEP declarations of Indel types can be found:
*** SNPEff: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/snpEff/*
*** VEP: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/VEP/*
** We used an '''Indel filtering strategy''' composed of many levels.
**# AC must be 1 or greater -- eliminate Indels with AC=0
**# the Indel should overlap with a VNTR region or overlap with another Indel. We used Adrian's annotate indels program to identify such overlaps. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/All.annotated.sites.vcf.gz
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/Overlaps.txt
**# at least 50% of the Indels should have informative AD field (we define "informative" to mean that the sample has U/(R+A+U)<0.50). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter2_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# at least 50% of the Indels should have informative PL field (we define "informative" to mean that the PL field for the sample is anything BUT ././. or 0/0/0). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter1_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# the Indel needs to have BF_LRE_LUD (a Bayes factor comparing a. related & HWE to b. unrelated & HWD) > -10. Higher BF_LRE_LUD should indicate a better Indel. We used Hyun's MiLK program to obtain BF_LRE_LUD values. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/all.genotypes.milk.sites.vcf
*** Overall results from all of the below filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/FinalIndelFilteringStatistics.txt
*** VCFs of Indels after filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.*.PASS.vcf.gz
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.PASS.vcf.gz

* Merged Indel VCF with beagles SNP VCF, then sorted to get the following VCFS:
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/chr*.PASS.beagled.vcf.gz

* Indel and SNP VCFs that have been combined AND BEAGLED AGAIN TOGETHER using Beagle4. '''These are the latest VCFS''':
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/beagle4/beagle4_chr*/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''mtDNA Copy Number''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/sardiniaCopyNumber_include.txt
** By sample copy number results by sample can be found: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/*.CopyNumber.noRand.500000.*.txt

* '''Filtering Samples''': Discrepancy Figures
** We compared chip data from previous work to the sequencing data produced now in hopes of identifying which samples have reliable sequencing data. This comparison was done for each chromosome separately and then combined into an overall discrepancy figure. Only 3189 of the 3840 samples (3188 of 3839 if you throw out sample with two sets of data) had
** By chromosome discrepancy figures can be found:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/*.diff.discordance_matrix

=== What is Complete ===
* SNP Call
** 24,901,469 SNPs passed filters
*** 16,822,922 are in dbSNP (67.6%)
*** %Known Ts/Tv - 2.24
*** %Novel Ts/Tv - 1.95
* InDel Call
** 1,194,945 passed filters

=== Future Directions ===
* '''Sample Filtering'''
** We did not do any filtering of samples (based on dupRate, genome coverage, mapping rate, proper paired, mean depth, or any other QPLOT stats) prior to SNP and Indel calling. Because of this, we want to do this filtering now. 3,188 or 3,839 samples have genome chip data from a few years ago. For these, we could look at the non-reference concordance between the chip genotypes and the sequencing genotypes and declare 'bad' samples to be those that fall below a certain threshold, such as 98% non-ref concordance. However, since the remaining 651 samples do not have chip data, this is not an option for them. Therefore, we decided on the following strategy instead:
**# Calculate non-reference concordance for the 3,188 samples that have chip data.
**# Create a prediction model using QPLOT statistics as predictors of non-reference concordance. Either do so on all of the 3,188 samples and look at R^2 (likely inflated from overfitting) or use cross-validation (test and training set) to give a measure of external predictive power.
**# If reasonable predictive power/R^2, use the prediction model to estimate the non-reference concordance amongst the 651 samples that do not have chip data. Also use the prediction model to estimate the non-reference concordance among the 3,188 samples that do have chip data.
**# Set a cut-off for 'good' versus 'bad' samples based on the estimated non-reference concordance and use it to filter samples.

== Key References ==

If you are looking to learn about the project, I strongly recommend that you read the following papers

* Pilia G, Chen WM, Scuteri A, Orru M, Albai G, Dei M, Lai S, Usala G, Lai M, Loi P, Mameli C, Vacca L, Deiana M, Olla N, Masala M, Cao A, Najjar SS, Terracciano A, Nedorezov T, Sharov A, Zonderman AB, Abecasis GR, Costa P, Lakatta E and Schlessinger D (2006). Heritability of Cardiovascular and Personality Traits in 6,148 Sardinians. PLoS Genet 2:1207-1223 [[http://www.sph.umich.edu/csg/abecasis/publications/16934002.html Abstract and PDF]]

* Scuteri A, Sanna S, Chen WM, Uda M, Albai G, Strait J, Najjar S, Nagarajah R, Orru M, Usala G, Dei M, Lai S, Maschio A, Busonero F, Mulas A, Ehret GB, Fink AA, Weder A, Cooper R, Galan P, Chakravarti A, Schlessinger D, Cao A, Lakatta E and Abecasis GR (2007). Genome Wide Association Scan shows Genetic Variants in the FTO gene are Associated with Obesity Related Traits PLoS Genetics 3:1200-10 [[http://www.sph.umich.edu/csg/abecasis/publications/PLOS-Obesity-Scan.html Abstract and PDF]]

* Chen WM and Abecasis GR (2007). Family-based association tests for genomewide association scans. Am J Hum Genet 81:913-26 [[http://www.sph.umich.edu/csg/abecasis/publications/17924335.html Abstract and PDF]]

SardiNIA

2016-07-06T15:55:04Z

Kleckner: /* Future Directions */

== Project Leaders ==

* David Schlessinger (National Institutes on Aging, Baltimore)
* Manuela Uda (National Research Center, Cagliari, Italy)
* Goncalo Abecasis (University of Michigan, Ann Arbor)

== Genomewide Association Study ==

We carried out an initial genomewide association by genotyping 1405 samples with Affymetrix 500K SNP arrays. Then, because SardiNIA samples are closely related to each other, we were able to use these genotypes to impute the genomes of many close relatives who were genotyped with Affymetrix 10K SNP arrays. Typically, our SardiNIA GWAS analysis thus include approximately 4300 genotyped or imputed individuals. To learn more about the approach see Chen and Abecasis (2007) and Scuteri et al (2007).

=== Planned Updates ===

Several ongoing experiments are expected to gradually improve our GWAS data. First, we expect to integrate Affymetrix 1M SNP chip genotypes into our analyses. Second, we plan to genotype all sampled individuals with the Metabochip, which includes 200,000 SNPs. This will enable us to fill in genotypes for relatives of those genotyped with denser arrays more accurately, because we will more precisely identify shared stretches of chromosome.

== Medical Sequencing Project ==

We are sequencing the genomes of 1,000 individuals to learn about the genetics of blood lipid levels and personality.

== Status as of July 2016 ==

=== Locations of Files for Current Data Freeze of 3839 Samples===

NOTICE: We identified late in the process that two of the samples (22855 and 22385) were actually the same individual. They both should be the same individual 22855. Therefore, there are 3840 sample IDs in each of the files below, but only 22855 should move on to later processes. In future data freezes with this data, these two sequencing sets should be merged into a single 22855 individual.

* List of '''Sample Numbers'''
** The following file contains three columns: [SampleID used in these analyses] [ID supplied by CSCT or Sardinia or other project] [Sequencing core ID (if different)]:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/sampleIDConversion.txt

* List of paths to '''BAMs''' used in this data freeze (Index file)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_index_20150504.index
** 401 of these samples have some new BAM contribution since the previous data freeze... their BAMs can be found here: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/newsamples.index

* '''Pedigree''' (Not too helpful -- used for SNPCall)
** All Samples: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_ped_20150510.ped
** Disjoint Trios: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/Pedigree_Fall15_DataFreeze_Triplets.ped

* '''QC''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCStats.txt
** For a list of paths to all of the separate QPLOT files for each sample, see the file: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCFileListFinal.txt

* '''SNPCall''' Results (Produced with Gotcloud SNPCall and phased using Beagle4)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''IndelCall''' Results (Produced with Gotcloud Indel)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/indel/final/all.genotypes.sites.vcf.gz
** SNPEff and VEP declarations of Indel types can be found:
*** SNPEff: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/snpEff/*
*** VEP: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/VEP/*
** We used an '''Indel filtering strategy''' composed of many levels.
**# AC must be 1 or greater -- eliminate Indels with AC=0
**# the Indel should overlap with a VNTR region or overlap with another Indel. We used Adrian's annotate indels program to identify such overlaps. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/All.annotated.sites.vcf.gz
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/Overlaps.txt
**# at least 50% of the Indels should have informative AD field (we define "informative" to mean that the sample has U/(R+A+U)<0.50). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter2_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# at least 50% of the Indels should have informative PL field (we define "informative" to mean that the PL field for the sample is anything BUT ././. or 0/0/0). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter1_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# the Indel needs to have BF_LRE_LUD (a Bayes factor comparing a. related & HWE to b. unrelated & HWD) > -10. Higher BF_LRE_LUD should indicate a better Indel. We used Hyun's MiLK program to obtain BF_LRE_LUD values. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/all.genotypes.milk.sites.vcf
*** Overall results from all of the below filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/FinalIndelFilteringStatistics.txt
*** VCFs of Indels after filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.*.PASS.vcf.gz
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.PASS.vcf.gz

* Merged Indel VCF with beagles SNP VCF, then sorted to get the following VCFS:
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/chr*.PASS.beagled.vcf.gz

* Indel and SNP VCFs that have been combined AND BEAGLED AGAIN TOGETHER using Beagle4. '''These are the latest VCFS''':
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/beagle4/beagle4_chr*/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''mtDNA Copy Number''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/sardiniaCopyNumber_include.txt
** By sample copy number results by sample can be found: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/*.CopyNumber.noRand.500000.*.txt

* '''Filtering Samples''': Discrepancy Figures
** We compared chip data from previous work to the sequencing data produced now in hopes of identifying which samples have reliable sequencing data. This comparison was done for each chromosome separately and then combined into an overall discrepancy figure. Only 3189 of the 3840 samples (3188 of 3839 if you throw out sample with two sets of data) had
** By chromosome discrepancy figures can be found:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/*.diff.discordance_matrix

=== What is Complete ===
* SNP Call
** 24,901,469 SNPs passed filters
*** 16,822,922 are in dbSNP (67.6%)
*** %Known Ts/Tv - 2.24
*** %Novel Ts/Tv - 1.95
* InDel Call
** 1,194,945 passed filters

=== Future Directions ===
* Sample Filtering
** We did not do any filtering of samples (based on dupRate, genome coverage, mapping rate, proper paired, mean depth, or any other QPLOT stats) prior to SNP and Indel calling. Because of this, we want to do this filtering now. 3,188 or 3,839 samples have genome chip data from a few years ago. For these, we could look at the non-reference concordance between the chip genotypes and the sequencing genotypes and declare 'bad' samples to be those that fall below a certain threshold, such as 98% non-ref concordance. However, since the remaining 651 samples do not have chip data, this is not an option for them. Therefore, we decided on the following strategy instead:
**# Calculate non-reference concordance for the 3,188 samples that have chip data.
**# Create a prediction model using QPLOT statistics as predictors of non-reference concordance. Either do so on all of the 3,188 samples and look at R^2 (likely inflated from overfitting) or use cross-validation (test and training set) to give a measure of external predictive power.
**# If reasonable predictive power/R^2, use the prediction model to estimate the non-reference concordance amongst the 651 samples that do not have chip data. Also use the prediction model to estimate the non-reference concordance among the 3,188 samples that do have chip data.
**# Set a cut-off for 'good' versus 'bad' samples based on the estimated non-reference concordance and use it to filter samples.

== Key References ==

If you are looking to learn about the project, I strongly recommend that you read the following papers

* Pilia G, Chen WM, Scuteri A, Orru M, Albai G, Dei M, Lai S, Usala G, Lai M, Loi P, Mameli C, Vacca L, Deiana M, Olla N, Masala M, Cao A, Najjar SS, Terracciano A, Nedorezov T, Sharov A, Zonderman AB, Abecasis GR, Costa P, Lakatta E and Schlessinger D (2006). Heritability of Cardiovascular and Personality Traits in 6,148 Sardinians. PLoS Genet 2:1207-1223 [[http://www.sph.umich.edu/csg/abecasis/publications/16934002.html Abstract and PDF]]

* Scuteri A, Sanna S, Chen WM, Uda M, Albai G, Strait J, Najjar S, Nagarajah R, Orru M, Usala G, Dei M, Lai S, Maschio A, Busonero F, Mulas A, Ehret GB, Fink AA, Weder A, Cooper R, Galan P, Chakravarti A, Schlessinger D, Cao A, Lakatta E and Abecasis GR (2007). Genome Wide Association Scan shows Genetic Variants in the FTO gene are Associated with Obesity Related Traits PLoS Genetics 3:1200-10 [[http://www.sph.umich.edu/csg/abecasis/publications/PLOS-Obesity-Scan.html Abstract and PDF]]

* Chen WM and Abecasis GR (2007). Family-based association tests for genomewide association scans. Am J Hum Genet 81:913-26 [[http://www.sph.umich.edu/csg/abecasis/publications/17924335.html Abstract and PDF]]

SardiNIA

2016-07-06T15:44:56Z

Kleckner: /* What is Complete */

== Project Leaders ==

* David Schlessinger (National Institutes on Aging, Baltimore)
* Manuela Uda (National Research Center, Cagliari, Italy)
* Goncalo Abecasis (University of Michigan, Ann Arbor)

== Genomewide Association Study ==

We carried out an initial genomewide association by genotyping 1405 samples with Affymetrix 500K SNP arrays. Then, because SardiNIA samples are closely related to each other, we were able to use these genotypes to impute the genomes of many close relatives who were genotyped with Affymetrix 10K SNP arrays. Typically, our SardiNIA GWAS analysis thus include approximately 4300 genotyped or imputed individuals. To learn more about the approach see Chen and Abecasis (2007) and Scuteri et al (2007).

=== Planned Updates ===

Several ongoing experiments are expected to gradually improve our GWAS data. First, we expect to integrate Affymetrix 1M SNP chip genotypes into our analyses. Second, we plan to genotype all sampled individuals with the Metabochip, which includes 200,000 SNPs. This will enable us to fill in genotypes for relatives of those genotyped with denser arrays more accurately, because we will more precisely identify shared stretches of chromosome.

== Medical Sequencing Project ==

We are sequencing the genomes of 1,000 individuals to learn about the genetics of blood lipid levels and personality.

== Status as of July 2016 ==

=== Locations of Files for Current Data Freeze of 3839 Samples===

NOTICE: We identified late in the process that two of the samples (22855 and 22385) were actually the same individual. They both should be the same individual 22855. Therefore, there are 3840 sample IDs in each of the files below, but only 22855 should move on to later processes. In future data freezes with this data, these two sequencing sets should be merged into a single 22855 individual.

* List of '''Sample Numbers'''
** The following file contains three columns: [SampleID used in these analyses] [ID supplied by CSCT or Sardinia or other project] [Sequencing core ID (if different)]:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/sampleIDConversion.txt

* List of paths to '''BAMs''' used in this data freeze (Index file)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_index_20150504.index
** 401 of these samples have some new BAM contribution since the previous data freeze... their BAMs can be found here: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/newsamples.index

* '''Pedigree''' (Not too helpful -- used for SNPCall)
** All Samples: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_ped_20150510.ped
** Disjoint Trios: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/Pedigree_Fall15_DataFreeze_Triplets.ped

* '''QC''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCStats.txt
** For a list of paths to all of the separate QPLOT files for each sample, see the file: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCFileListFinal.txt

* '''SNPCall''' Results (Produced with Gotcloud SNPCall and phased using Beagle4)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''IndelCall''' Results (Produced with Gotcloud Indel)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/indel/final/all.genotypes.sites.vcf.gz
** SNPEff and VEP declarations of Indel types can be found:
*** SNPEff: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/snpEff/*
*** VEP: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/VEP/*
** We used an '''Indel filtering strategy''' composed of many levels.
**# AC must be 1 or greater -- eliminate Indels with AC=0
**# the Indel should overlap with a VNTR region or overlap with another Indel. We used Adrian's annotate indels program to identify such overlaps. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/All.annotated.sites.vcf.gz
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/Overlaps.txt
**# at least 50% of the Indels should have informative AD field (we define "informative" to mean that the sample has U/(R+A+U)<0.50). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter2_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# at least 50% of the Indels should have informative PL field (we define "informative" to mean that the PL field for the sample is anything BUT ././. or 0/0/0). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter1_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# the Indel needs to have BF_LRE_LUD (a Bayes factor comparing a. related & HWE to b. unrelated & HWD) > -10. Higher BF_LRE_LUD should indicate a better Indel. We used Hyun's MiLK program to obtain BF_LRE_LUD values. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/all.genotypes.milk.sites.vcf
*** Overall results from all of the below filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/FinalIndelFilteringStatistics.txt
*** VCFs of Indels after filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.*.PASS.vcf.gz
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.PASS.vcf.gz

* Merged Indel VCF with beagles SNP VCF, then sorted to get the following VCFS:
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/chr*.PASS.beagled.vcf.gz

* Indel and SNP VCFs that have been combined AND BEAGLED AGAIN TOGETHER using Beagle4. '''These are the latest VCFS''':
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/beagle4/beagle4_chr*/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''mtDNA Copy Number''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/sardiniaCopyNumber_include.txt
** By sample copy number results by sample can be found: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/*.CopyNumber.noRand.500000.*.txt

* '''Filtering Samples''': Discrepancy Figures
** We compared chip data from previous work to the sequencing data produced now in hopes of identifying which samples have reliable sequencing data. This comparison was done for each chromosome separately and then combined into an overall discrepancy figure. Only 3189 of the 3840 samples (3188 of 3839 if you throw out sample with two sets of data) had
** By chromosome discrepancy figures can be found:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/*.diff.discordance_matrix

=== What is Complete ===
* SNP Call
** 24,901,469 SNPs passed filters
*** 16,822,922 are in dbSNP (67.6%)
*** %Known Ts/Tv - 2.24
*** %Novel Ts/Tv - 1.95
* InDel Call
** 1,194,945 passed filters

=== Future Directions ===

== Key References ==

If you are looking to learn about the project, I strongly recommend that you read the following papers

* Pilia G, Chen WM, Scuteri A, Orru M, Albai G, Dei M, Lai S, Usala G, Lai M, Loi P, Mameli C, Vacca L, Deiana M, Olla N, Masala M, Cao A, Najjar SS, Terracciano A, Nedorezov T, Sharov A, Zonderman AB, Abecasis GR, Costa P, Lakatta E and Schlessinger D (2006). Heritability of Cardiovascular and Personality Traits in 6,148 Sardinians. PLoS Genet 2:1207-1223 [[http://www.sph.umich.edu/csg/abecasis/publications/16934002.html Abstract and PDF]]

* Scuteri A, Sanna S, Chen WM, Uda M, Albai G, Strait J, Najjar S, Nagarajah R, Orru M, Usala G, Dei M, Lai S, Maschio A, Busonero F, Mulas A, Ehret GB, Fink AA, Weder A, Cooper R, Galan P, Chakravarti A, Schlessinger D, Cao A, Lakatta E and Abecasis GR (2007). Genome Wide Association Scan shows Genetic Variants in the FTO gene are Associated with Obesity Related Traits PLoS Genetics 3:1200-10 [[http://www.sph.umich.edu/csg/abecasis/publications/PLOS-Obesity-Scan.html Abstract and PDF]]

* Chen WM and Abecasis GR (2007). Family-based association tests for genomewide association scans. Am J Hum Genet 81:913-26 [[http://www.sph.umich.edu/csg/abecasis/publications/17924335.html Abstract and PDF]]

SardiNIA

2016-07-06T15:37:03Z

Kleckner:

== Project Leaders ==

* David Schlessinger (National Institutes on Aging, Baltimore)
* Manuela Uda (National Research Center, Cagliari, Italy)
* Goncalo Abecasis (University of Michigan, Ann Arbor)

== Genomewide Association Study ==

We carried out an initial genomewide association by genotyping 1405 samples with Affymetrix 500K SNP arrays. Then, because SardiNIA samples are closely related to each other, we were able to use these genotypes to impute the genomes of many close relatives who were genotyped with Affymetrix 10K SNP arrays. Typically, our SardiNIA GWAS analysis thus include approximately 4300 genotyped or imputed individuals. To learn more about the approach see Chen and Abecasis (2007) and Scuteri et al (2007).

=== Planned Updates ===

Several ongoing experiments are expected to gradually improve our GWAS data. First, we expect to integrate Affymetrix 1M SNP chip genotypes into our analyses. Second, we plan to genotype all sampled individuals with the Metabochip, which includes 200,000 SNPs. This will enable us to fill in genotypes for relatives of those genotyped with denser arrays more accurately, because we will more precisely identify shared stretches of chromosome.

== Medical Sequencing Project ==

We are sequencing the genomes of 1,000 individuals to learn about the genetics of blood lipid levels and personality.

== Status as of July 2016 ==

=== Locations of Files for Current Data Freeze of 3839 Samples===

NOTICE: We identified late in the process that two of the samples (22855 and 22385) were actually the same individual. They both should be the same individual 22855. Therefore, there are 3840 sample IDs in each of the files below, but only 22855 should move on to later processes. In future data freezes with this data, these two sequencing sets should be merged into a single 22855 individual.

* List of '''Sample Numbers'''
** The following file contains three columns: [SampleID used in these analyses] [ID supplied by CSCT or Sardinia or other project] [Sequencing core ID (if different)]:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/sampleIDConversion.txt

* List of paths to '''BAMs''' used in this data freeze (Index file)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_index_20150504.index
** 401 of these samples have some new BAM contribution since the previous data freeze... their BAMs can be found here: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/newsamples.index

* '''Pedigree''' (Not too helpful -- used for SNPCall)
** All Samples: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_ped_20150510.ped
** Disjoint Trios: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/Pedigree_Fall15_DataFreeze_Triplets.ped

* '''QC''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCStats.txt
** For a list of paths to all of the separate QPLOT files for each sample, see the file: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCFileListFinal.txt

* '''SNPCall''' Results (Produced with Gotcloud SNPCall and phased using Beagle4)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''IndelCall''' Results (Produced with Gotcloud Indel)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/indel/final/all.genotypes.sites.vcf.gz
** SNPEff and VEP declarations of Indel types can be found:
*** SNPEff: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/snpEff/*
*** VEP: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/VEP/*
** We used an '''Indel filtering strategy''' composed of many levels.
**# AC must be 1 or greater -- eliminate Indels with AC=0
**# the Indel should overlap with a VNTR region or overlap with another Indel. We used Adrian's annotate indels program to identify such overlaps. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/All.annotated.sites.vcf.gz
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/Overlaps.txt
**# at least 50% of the Indels should have informative AD field (we define "informative" to mean that the sample has U/(R+A+U)<0.50). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter2_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# at least 50% of the Indels should have informative PL field (we define "informative" to mean that the PL field for the sample is anything BUT ././. or 0/0/0). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter1_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# the Indel needs to have BF_LRE_LUD (a Bayes factor comparing a. related & HWE to b. unrelated & HWD) > -10. Higher BF_LRE_LUD should indicate a better Indel. We used Hyun's MiLK program to obtain BF_LRE_LUD values. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/all.genotypes.milk.sites.vcf
*** Overall results from all of the below filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/FinalIndelFilteringStatistics.txt
*** VCFs of Indels after filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.*.PASS.vcf.gz
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.PASS.vcf.gz

* Merged Indel VCF with beagles SNP VCF, then sorted to get the following VCFS:
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/chr*.PASS.beagled.vcf.gz

* Indel and SNP VCFs that have been combined AND BEAGLED AGAIN TOGETHER using Beagle4. '''These are the latest VCFS''':
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/beagle4/beagle4_chr*/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''mtDNA Copy Number''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/sardiniaCopyNumber_include.txt
** By sample copy number results by sample can be found: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/*.CopyNumber.noRand.500000.*.txt

* '''Filtering Samples''': Discrepancy Figures
** We compared chip data from previous work to the sequencing data produced now in hopes of identifying which samples have reliable sequencing data. This comparison was done for each chromosome separately and then combined into an overall discrepancy figure. Only 3189 of the 3840 samples (3188 of 3839 if you throw out sample with two sets of data) had
** By chromosome discrepancy figures can be found:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/*.diff.discordance_matrix

=== What is Complete ===

=== Future Directions ===

== Key References ==

If you are looking to learn about the project, I strongly recommend that you read the following papers

* Pilia G, Chen WM, Scuteri A, Orru M, Albai G, Dei M, Lai S, Usala G, Lai M, Loi P, Mameli C, Vacca L, Deiana M, Olla N, Masala M, Cao A, Najjar SS, Terracciano A, Nedorezov T, Sharov A, Zonderman AB, Abecasis GR, Costa P, Lakatta E and Schlessinger D (2006). Heritability of Cardiovascular and Personality Traits in 6,148 Sardinians. PLoS Genet 2:1207-1223 [[http://www.sph.umich.edu/csg/abecasis/publications/16934002.html Abstract and PDF]]

* Scuteri A, Sanna S, Chen WM, Uda M, Albai G, Strait J, Najjar S, Nagarajah R, Orru M, Usala G, Dei M, Lai S, Maschio A, Busonero F, Mulas A, Ehret GB, Fink AA, Weder A, Cooper R, Galan P, Chakravarti A, Schlessinger D, Cao A, Lakatta E and Abecasis GR (2007). Genome Wide Association Scan shows Genetic Variants in the FTO gene are Associated with Obesity Related Traits PLoS Genetics 3:1200-10 [[http://www.sph.umich.edu/csg/abecasis/publications/PLOS-Obesity-Scan.html Abstract and PDF]]

* Chen WM and Abecasis GR (2007). Family-based association tests for genomewide association scans. Am J Hum Genet 81:913-26 [[http://www.sph.umich.edu/csg/abecasis/publications/17924335.html Abstract and PDF]]

SardiNIA

2016-06-27T16:42:48Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

== Project Leaders ==

* David Schlessinger (National Institutes on Aging, Baltimore)
* Manuela Uda (National Research Center, Cagliari, Italy)
* Goncalo Abecasis (University of Michigan, Ann Arbor)

== Genomewide Association Study ==

We carried out an initial genomewide association by genotyping 1405 samples with Affymetrix 500K SNP arrays. Then, because SardiNIA samples are closely related to each other, we were able to use these genotypes to impute the genomes of many close relatives who were genotyped with Affymetrix 10K SNP arrays. Typically, our SardiNIA GWAS analysis thus include approximately 4300 genotyped or imputed individuals. To learn more about the approach see Chen and Abecasis (2007) and Scuteri et al (2007).

=== Planned Updates ===

Several ongoing experiments are expected to gradually improve our GWAS data. First, we expect to integrate Affymetrix 1M SNP chip genotypes into our analyses. Second, we plan to genotype all sampled individuals with the Metabochip, which includes 200,000 SNPs. This will enable us to fill in genotypes for relatives of those genotyped with denser arrays more accurately, because we will more precisely identify shared stretches of chromosome.

== Medical Sequencing Project ==

We are sequencing the genomes of 1,000 individuals to learn about the genetics of blood lipid levels and personality.

== Status as of June 2016 ==

=== Locations of Files for Current Data Freeze of 3839 Samples===

NOTICE: We identified late in the process that two of the samples (22855 and 22385) were actually the same individual. They both should be the same individual 22855. Therefore, there are 3840 sample IDs in each of the files below, but only 22855 should move on to later processes. In future data freezes with this data, these two sequencing sets should be merged into a single 22855 individual.

* List of '''Sample Numbers'''
** The following file contains three columns: [SampleID used in these analyses] [ID supplied by CSCT or Sardinia or other project] [Sequencing core ID (if different)]:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/sampleIDConversion.txt

* List of paths to '''BAMs''' used in this data freeze (Index file)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_index_20150504.index
** 401 of these samples have some new BAM contribution since the previous data freeze... their BAMs can be found here: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/newsamples.index

* '''Pedigree''' (Not too helpful -- used for SNPCall)
** All Samples: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_ped_20150510.ped
** Disjoint Trios: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/Pedigree_Fall15_DataFreeze_Triplets.ped

* '''QC''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCStats.txt
** For a list of paths to all of the separate QPLOT files for each sample, see the file: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCFileListFinal.txt

* '''SNPCall''' Results (Produced with Gotcloud SNPCall and phased using Beagle4)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''IndelCall''' Results (Produced with Gotcloud Indel)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/indel/final/all.genotypes.sites.vcf.gz
** SNPEff and VEP declarations of Indel types can be found:
*** SNPEff: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/snpEff/*
*** VEP: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/VEP/*
** We used an '''Indel filtering strategy''' composed of many levels.
**# AC must be 1 or greater -- eliminate Indels with AC=0
**# the Indel should overlap with a VNTR region or overlap with another Indel. We used Adrian's annotate indels program to identify such overlaps. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/All.annotated.sites.vcf.gz
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/Overlaps.txt
**# at least 50% of the Indels should have informative AD field (we define "informative" to mean that the sample has U/(R+A+U)<0.50). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter2_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# at least 50% of the Indels should have informative PL field (we define "informative" to mean that the PL field for the sample is anything BUT ././. or 0/0/0). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter1_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# the Indel needs to have BF_LRE_LUD (a Bayes factor comparing a. related & HWE to b. unrelated & HWD) > -10. Higher BF_LRE_LUD should indicate a better Indel. We used Hyun's MiLK program to obtain BF_LRE_LUD values. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/all.genotypes.milk.sites.vcf
*** Overall results from all of the below filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/FinalIndelFilteringStatistics.txt
*** VCFs of Indels after filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.*.PASS.vcf.gz
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.PASS.vcf.gz

* Merged Indel VCF with beagles SNP VCF, then sorted to get the following VCFS:
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/chr*.PASS.beagled.vcf.gz

* Indel and SNP VCFs that have been combined AND BEAGLED AGAIN TOGETHER using Beagle4. '''These are the latest VCFS''':
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/beagle4/beagle4_chr*/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''mtDNA Copy Number''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/sardiniaCopyNumber_include.txt
** By sample copy number results by sample can be found: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/*.CopyNumber.noRand.500000.*.txt

* '''Filtering Samples''': Discrepancy Figures
** We compared chip data from previous work to the sequencing data produced now in hopes of identifying which samples have reliable sequencing data. This comparison was done for each chromosome separately and then combined into an overall discrepancy figure. Only 3189 of the 3840 samples (3188 of 3839 if you throw out sample with two sets of data) had
** By chromosome discrepancy figures can be found:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/*.diff.discordance_matrix
** Overall discrepancy can be found:
*** '''''IN PROCESS'''''

=== What is Complete ===

=== Future Directions ===

== Key References ==

If you are looking to learn about the project, I strongly recommend that you read the following papers

* Pilia G, Chen WM, Scuteri A, Orru M, Albai G, Dei M, Lai S, Usala G, Lai M, Loi P, Mameli C, Vacca L, Deiana M, Olla N, Masala M, Cao A, Najjar SS, Terracciano A, Nedorezov T, Sharov A, Zonderman AB, Abecasis GR, Costa P, Lakatta E and Schlessinger D (2006). Heritability of Cardiovascular and Personality Traits in 6,148 Sardinians. PLoS Genet 2:1207-1223 [[http://www.sph.umich.edu/csg/abecasis/publications/16934002.html Abstract and PDF]]

* Scuteri A, Sanna S, Chen WM, Uda M, Albai G, Strait J, Najjar S, Nagarajah R, Orru M, Usala G, Dei M, Lai S, Maschio A, Busonero F, Mulas A, Ehret GB, Fink AA, Weder A, Cooper R, Galan P, Chakravarti A, Schlessinger D, Cao A, Lakatta E and Abecasis GR (2007). Genome Wide Association Scan shows Genetic Variants in the FTO gene are Associated with Obesity Related Traits PLoS Genetics 3:1200-10 [[http://www.sph.umich.edu/csg/abecasis/publications/PLOS-Obesity-Scan.html Abstract and PDF]]

* Chen WM and Abecasis GR (2007). Family-based association tests for genomewide association scans. Am J Hum Genet 81:913-26 [[http://www.sph.umich.edu/csg/abecasis/publications/17924335.html Abstract and PDF]]

SardiNIA

2016-06-27T16:41:54Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

== Project Leaders ==

* David Schlessinger (National Institutes on Aging, Baltimore)
* Manuela Uda (National Research Center, Cagliari, Italy)
* Goncalo Abecasis (University of Michigan, Ann Arbor)

== Genomewide Association Study ==

We carried out an initial genomewide association by genotyping 1405 samples with Affymetrix 500K SNP arrays. Then, because SardiNIA samples are closely related to each other, we were able to use these genotypes to impute the genomes of many close relatives who were genotyped with Affymetrix 10K SNP arrays. Typically, our SardiNIA GWAS analysis thus include approximately 4300 genotyped or imputed individuals. To learn more about the approach see Chen and Abecasis (2007) and Scuteri et al (2007).

=== Planned Updates ===

Several ongoing experiments are expected to gradually improve our GWAS data. First, we expect to integrate Affymetrix 1M SNP chip genotypes into our analyses. Second, we plan to genotype all sampled individuals with the Metabochip, which includes 200,000 SNPs. This will enable us to fill in genotypes for relatives of those genotyped with denser arrays more accurately, because we will more precisely identify shared stretches of chromosome.

== Medical Sequencing Project ==

We are sequencing the genomes of 1,000 individuals to learn about the genetics of blood lipid levels and personality.

== Status as of June 2016 ==

=== Locations of Files for Current Data Freeze of 3839 Samples===

NOTICE: We identified late in the process that two of the samples (22855 and 22385) were actually the same individual. They both should be the same individual 22855. Therefore, there are 3840 sample IDs in each of the files below, but only 22855 should move on to later processes. In future data freezes with this data, these two sequencing sets should be merged into a single 22855 individual.

* List of '''Sample Numbers'''
** The following file contains three columns: [SampleID used in these analyses] [ID supplied by CSCT or Sardinia or other project] [Sequencing core ID (if different)]:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/sampleIDConversion.txt

* List of paths to '''BAMs''' used in this data freeze (Index file)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_index_20150504.index
** 401 of these samples have some new BAM contribution since the previous data freeze... their BAMs can be found here: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/newsamples.index

* '''Pedigree''' (Not too helpful -- used for SNPCall)
** All Samples: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_ped_20150510.ped
** Disjoint Trios: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/Pedigree_Fall15_DataFreeze_Triplets.ped

* '''QC''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCStats.txt
** For a list of paths to all of the separate QPLOT files for each sample, see the file: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCFileListFinal.txt

* '''SNPCall''' Results (Produced with Gotcloud SNPCall and phased using Beagle4)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''IndelCall''' Results (Produced with Gotcloud Indel)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/indel/final/all.genotypes.sites.vcf.gz
** SNPEff and VEP declarations of Indel types can be found:
*** SNPEff: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/snpEff/*
*** VEP: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/VEP/*
** We used an '''Indel filtering strategy''' composed of many levels.
**# AC must be 1 or greater -- eliminate Indels with AC=0
**# the Indel should overlap with a VNTR region or overlap with another Indel. We used Adrian's annotate indels program to identify such overlaps. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/All.annotated.sites.vcf.gz
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/Overlaps.txt
**# at least 50% of the Indels should have informative AD field (we define "informative" to mean that the sample has U/(R+A+U)<0.50). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter2_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# at least 50% of the Indels should have informative PL field (we define "informative" to mean that the PL field for the sample is anything BUT ././. or 0/0/0). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter1_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# the Indel needs to have BF_LRE_LUD (a Bayes factor comparing a. related & HWE to b. unrelated & HWD) > -10. Higher BF_LRE_LUD should indicate a better Indel. We used Hyun's MiLK program to obtain BF_LRE_LUD values. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/all.genotypes.milk.sites.vcf
*** Overall results from all of the below filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/FinalIndelFilteringStatistics.txt
*** VCFs of Indels after filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.*.PASS.vcf.gz
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.PASS.vcf.gz

* Merged Indel VCF with beagles SNP VCF, then sorted to get the following VCFS:
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/chr*.PASS.beagled.vcf.gz

* Indel and SNP VCFs that have been combined AND BEAGLED AGAIN TOGETHER using Beagle4. '''These are the latest VCFS''':
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/beagle4/beagle4_chr*/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''mtDNA Copy Number''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/sardiniaCopyNumber_include.txt
** By sample copy number results by sample can be found: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/*.CopyNumber.noRand.500000.*.txt

* '''Filtering Samples''': Discrepancy Figures
** We compared chip data from previous work to the sequencing data produced now in hopes of identifying which samples have reliable sequencing data. This comparison was done for each chromosome separately and then combined into an overall discrepancy figure. Only 3189 of the 3839 samples had
** By chromosome discrepancy figures can be found:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/*.diff.discordance_matrix
** Overall discrepancy can be found:
*** '''''IN PROCESS'''''

=== What is Complete ===

=== Future Directions ===

== Key References ==

If you are looking to learn about the project, I strongly recommend that you read the following papers

* Pilia G, Chen WM, Scuteri A, Orru M, Albai G, Dei M, Lai S, Usala G, Lai M, Loi P, Mameli C, Vacca L, Deiana M, Olla N, Masala M, Cao A, Najjar SS, Terracciano A, Nedorezov T, Sharov A, Zonderman AB, Abecasis GR, Costa P, Lakatta E and Schlessinger D (2006). Heritability of Cardiovascular and Personality Traits in 6,148 Sardinians. PLoS Genet 2:1207-1223 [[http://www.sph.umich.edu/csg/abecasis/publications/16934002.html Abstract and PDF]]

* Scuteri A, Sanna S, Chen WM, Uda M, Albai G, Strait J, Najjar S, Nagarajah R, Orru M, Usala G, Dei M, Lai S, Maschio A, Busonero F, Mulas A, Ehret GB, Fink AA, Weder A, Cooper R, Galan P, Chakravarti A, Schlessinger D, Cao A, Lakatta E and Abecasis GR (2007). Genome Wide Association Scan shows Genetic Variants in the FTO gene are Associated with Obesity Related Traits PLoS Genetics 3:1200-10 [[http://www.sph.umich.edu/csg/abecasis/publications/PLOS-Obesity-Scan.html Abstract and PDF]]

* Chen WM and Abecasis GR (2007). Family-based association tests for genomewide association scans. Am J Hum Genet 81:913-26 [[http://www.sph.umich.edu/csg/abecasis/publications/17924335.html Abstract and PDF]]

SardiNIA

2016-06-27T16:41:15Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

== Project Leaders ==

* David Schlessinger (National Institutes on Aging, Baltimore)
* Manuela Uda (National Research Center, Cagliari, Italy)
* Goncalo Abecasis (University of Michigan, Ann Arbor)

== Genomewide Association Study ==

We carried out an initial genomewide association by genotyping 1405 samples with Affymetrix 500K SNP arrays. Then, because SardiNIA samples are closely related to each other, we were able to use these genotypes to impute the genomes of many close relatives who were genotyped with Affymetrix 10K SNP arrays. Typically, our SardiNIA GWAS analysis thus include approximately 4300 genotyped or imputed individuals. To learn more about the approach see Chen and Abecasis (2007) and Scuteri et al (2007).

=== Planned Updates ===

Several ongoing experiments are expected to gradually improve our GWAS data. First, we expect to integrate Affymetrix 1M SNP chip genotypes into our analyses. Second, we plan to genotype all sampled individuals with the Metabochip, which includes 200,000 SNPs. This will enable us to fill in genotypes for relatives of those genotyped with denser arrays more accurately, because we will more precisely identify shared stretches of chromosome.

== Medical Sequencing Project ==

We are sequencing the genomes of 1,000 individuals to learn about the genetics of blood lipid levels and personality.

== Status as of June 2016 ==

=== Locations of Files for Current Data Freeze of 3839 Samples===

NOTICE: We identified late in the process that two of the samples (22855 and 22385) were actually the same individual. They both should be the same individual 22855. Therefore, there are 3840 sample IDs in each of the files below, but only 22855 should move on to later processes. In future data freezes with this data, these two sequencing sets should be merged into a single 22855 individual.

* List of '''Sample Numbers'''
** The following file contains three columns: [SampleID used in these analyses] [ID supplied by CSCT or Sardinia or other project] [Sequencing core ID (if different)]:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/sampleIDConversion.txt

* List of paths to '''BAMs''' used in this data freeze (Index file)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_index_20150504.index
** 401 of these samples have some new BAM contribution since the previous data freeze... their BAMs can be found here: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/newsamples.index

* '''Pedigree''' (Not too helpful -- used for SNPCall)
** All Samples: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_ped_20150510.ped
** Disjoint Trios: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/Pedigree_Fall15_DataFreeze_Triplets.ped

* '''QC''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCStats.txt
** For a list of paths to all of the separate QPLOT files for each sample, see the file: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCFileListFinal.txt

* '''SNPCall''' Results (Produced with Gotcloud SNPCall and phased using Beagle4)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''IndelCall''' Results (Produced with Gotcloud Indel)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/indel/final/all.genotypes.sites.vcf.gz
** SNPEff and VEP declarations of Indel types can be found:
*** SNPEff: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/snpEff/*
*** VEP: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/VEP/*
** We used an '''Indel filtering strategy''' composed of many levels.
**# AC must be 1 or greater -- eliminate Indels with AC=0
**# the Indel should overlap with a VNTR region or overlap with another Indel. We used Adrian's annotate indels program to identify such overlaps. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/All.annotated.sites.vcf.gz
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/Overlaps.txt
**# at least 50% of the Indels should have informative AD field (we define "informative" to mean that the sample has U/(R+A+U)<0.50). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter2_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# at least 50% of the Indels should have informative PL field (we define "informative" to mean that the PL field for the sample is anything BUT ././. or 0/0/0). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter1_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# the Indel needs to have BF_LRE_LUD (a Bayes factor comparing a. related & HWE to b. unrelated & HWD) > -10. Higher BF_LRE_LUD should indicate a better Indel. We used Hyun's MiLK program to obtain BF_LRE_LUD values. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/all.genotypes.milk.sites.vcf
*** Overall results from all of the below filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/FinalIndelFilteringStatistics.txt
*** VCFs of Indels after filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.*.PASS.vcf.gz
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.PASS.vcf.gz

* Merged Indel VCF with beagles SNP VCF, then sorted to get the following VCFS:
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/chr*.PASS.beagled.vcf.gz

* Indel and SNP VCFs that have been combined AND BEAGLED AGAIN TOGETHER using Beagle4. '''These are the latest VCFS''':
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/beagle4/beagle4_chr*/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''mtDNA Copy Number''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/sardiniaCopyNumber_include.txt
** By sample copy number results by sample can be found: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/*.CopyNumber.noRand.500000.*.txt

* '''Filtering Samples''': Discrepancy Figures
** We compared chip data from previous work to the sequencing data produced now in hopes of identifying which samples have reliable sequencing data. This comparison was done for each chromosome separately and then combined into an overall discrepancy figure. Only 3189 of the 3840 samples had
** By chromosome discrepancy figures can be found:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/*.diff.discordance_matrix
** Overall discrepancy can be found:
*** '''''IN PROCESS'''''

=== What is Complete ===

=== Future Directions ===

== Key References ==

If you are looking to learn about the project, I strongly recommend that you read the following papers

* Pilia G, Chen WM, Scuteri A, Orru M, Albai G, Dei M, Lai S, Usala G, Lai M, Loi P, Mameli C, Vacca L, Deiana M, Olla N, Masala M, Cao A, Najjar SS, Terracciano A, Nedorezov T, Sharov A, Zonderman AB, Abecasis GR, Costa P, Lakatta E and Schlessinger D (2006). Heritability of Cardiovascular and Personality Traits in 6,148 Sardinians. PLoS Genet 2:1207-1223 [[http://www.sph.umich.edu/csg/abecasis/publications/16934002.html Abstract and PDF]]

* Scuteri A, Sanna S, Chen WM, Uda M, Albai G, Strait J, Najjar S, Nagarajah R, Orru M, Usala G, Dei M, Lai S, Maschio A, Busonero F, Mulas A, Ehret GB, Fink AA, Weder A, Cooper R, Galan P, Chakravarti A, Schlessinger D, Cao A, Lakatta E and Abecasis GR (2007). Genome Wide Association Scan shows Genetic Variants in the FTO gene are Associated with Obesity Related Traits PLoS Genetics 3:1200-10 [[http://www.sph.umich.edu/csg/abecasis/publications/PLOS-Obesity-Scan.html Abstract and PDF]]

* Chen WM and Abecasis GR (2007). Family-based association tests for genomewide association scans. Am J Hum Genet 81:913-26 [[http://www.sph.umich.edu/csg/abecasis/publications/17924335.html Abstract and PDF]]

SardiNIA

2016-06-27T16:39:30Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

== Project Leaders ==

* David Schlessinger (National Institutes on Aging, Baltimore)
* Manuela Uda (National Research Center, Cagliari, Italy)
* Goncalo Abecasis (University of Michigan, Ann Arbor)

== Genomewide Association Study ==

We carried out an initial genomewide association by genotyping 1405 samples with Affymetrix 500K SNP arrays. Then, because SardiNIA samples are closely related to each other, we were able to use these genotypes to impute the genomes of many close relatives who were genotyped with Affymetrix 10K SNP arrays. Typically, our SardiNIA GWAS analysis thus include approximately 4300 genotyped or imputed individuals. To learn more about the approach see Chen and Abecasis (2007) and Scuteri et al (2007).

=== Planned Updates ===

Several ongoing experiments are expected to gradually improve our GWAS data. First, we expect to integrate Affymetrix 1M SNP chip genotypes into our analyses. Second, we plan to genotype all sampled individuals with the Metabochip, which includes 200,000 SNPs. This will enable us to fill in genotypes for relatives of those genotyped with denser arrays more accurately, because we will more precisely identify shared stretches of chromosome.

== Medical Sequencing Project ==

We are sequencing the genomes of 1,000 individuals to learn about the genetics of blood lipid levels and personality.

== Status as of June 2016 ==

=== Locations of Files for Current Data Freeze of 3839 Samples===

NOTICE: We identified late in the process that two of the samples (22855 and 22385) were actually the same individual. They both should be the same individual 22855. Therefore, there are 3840 sample IDs in each of the files below, but only 22855 should move on to later processes. In future data freezes with this data, these two sequencing sets should be merged into a single 22855 individual.

* List of '''Sample Numbers'''
** The following file contains three columns: [SampleID used in these analyses] [ID supplied by CSCT or Sardinia or other project] [Sequencing core ID (if different)]:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/sampleIDConversion.txt

* List of paths to '''BAMs''' used in this data freeze (Index file)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_index_20150504.index
** 401 of these samples have some new BAM contribution since the previous data freeze... their BAMs can be found here: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/newsamples.index

* '''Pedigree''' (Not too helpful -- used for SNPCall)
** All Samples: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_ped_20150510.ped

* '''QC''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCStats.txt
** For a list of paths to all of the separate QPLOT files for each sample, see the file: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCFileListFinal.txt

* '''SNPCall''' Results (Produced with Gotcloud SNPCall and phased using Beagle4)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''IndelCall''' Results (Produced with Gotcloud Indel)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/indel/final/all.genotypes.sites.vcf.gz
** SNPEff and VEP declarations of Indel types can be found:
*** SNPEff: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/snpEff/*
*** VEP: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/VEP/*
** We used an '''Indel filtering strategy''' composed of many levels.
**# AC must be 1 or greater -- eliminate Indels with AC=0
**# the Indel should overlap with a VNTR region or overlap with another Indel. We used Adrian's annotate indels program to identify such overlaps. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/All.annotated.sites.vcf.gz
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/Overlaps.txt
**# at least 50% of the Indels should have informative AD field (we define "informative" to mean that the sample has U/(R+A+U)<0.50). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter2_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# at least 50% of the Indels should have informative PL field (we define "informative" to mean that the PL field for the sample is anything BUT ././. or 0/0/0). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter1_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# the Indel needs to have BF_LRE_LUD (a Bayes factor comparing a. related & HWE to b. unrelated & HWD) > -10. Higher BF_LRE_LUD should indicate a better Indel. We used Hyun's MiLK program to obtain BF_LRE_LUD values. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/all.genotypes.milk.sites.vcf
*** Overall results from all of the below filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/FinalIndelFilteringStatistics.txt
*** VCFs of Indels after filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.*.PASS.vcf.gz
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.PASS.vcf.gz

* Merged Indel VCF with beagles SNP VCF, then sorted to get the following VCFS:
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/chr*.PASS.beagled.vcf.gz

* Indel and SNP VCFs that have been combined AND BEAGLED AGAIN TOGETHER using Beagle4. '''These are the latest VCFS''':
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/beagle4/beagle4_chr*/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''mtDNA Copy Number''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/sardiniaCopyNumber_include.txt
** By sample copy number results by sample can be found: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/*.CopyNumber.noRand.500000.*.txt

* '''Filtering Samples''': Discrepancy Figures
** We compared chip data from previous work to the sequencing data produced now in hopes of identifying which samples have reliable sequencing data. This comparison was done for each chromosome separately and then combined into an overall discrepancy figure. Only 3189 of the 3840 samples had
** By chromosome discrepancy figures can be found:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/*.diff.discordance_matrix
** Overall discrepancy can be found:
*** '''''IN PROCESS'''''

=== What is Complete ===

=== Future Directions ===

== Key References ==

If you are looking to learn about the project, I strongly recommend that you read the following papers

* Pilia G, Chen WM, Scuteri A, Orru M, Albai G, Dei M, Lai S, Usala G, Lai M, Loi P, Mameli C, Vacca L, Deiana M, Olla N, Masala M, Cao A, Najjar SS, Terracciano A, Nedorezov T, Sharov A, Zonderman AB, Abecasis GR, Costa P, Lakatta E and Schlessinger D (2006). Heritability of Cardiovascular and Personality Traits in 6,148 Sardinians. PLoS Genet 2:1207-1223 [[http://www.sph.umich.edu/csg/abecasis/publications/16934002.html Abstract and PDF]]

* Scuteri A, Sanna S, Chen WM, Uda M, Albai G, Strait J, Najjar S, Nagarajah R, Orru M, Usala G, Dei M, Lai S, Maschio A, Busonero F, Mulas A, Ehret GB, Fink AA, Weder A, Cooper R, Galan P, Chakravarti A, Schlessinger D, Cao A, Lakatta E and Abecasis GR (2007). Genome Wide Association Scan shows Genetic Variants in the FTO gene are Associated with Obesity Related Traits PLoS Genetics 3:1200-10 [[http://www.sph.umich.edu/csg/abecasis/publications/PLOS-Obesity-Scan.html Abstract and PDF]]

* Chen WM and Abecasis GR (2007). Family-based association tests for genomewide association scans. Am J Hum Genet 81:913-26 [[http://www.sph.umich.edu/csg/abecasis/publications/17924335.html Abstract and PDF]]

SardiNIA

2016-06-27T16:39:18Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

== Project Leaders ==

* David Schlessinger (National Institutes on Aging, Baltimore)
* Manuela Uda (National Research Center, Cagliari, Italy)
* Goncalo Abecasis (University of Michigan, Ann Arbor)

== Genomewide Association Study ==

We carried out an initial genomewide association by genotyping 1405 samples with Affymetrix 500K SNP arrays. Then, because SardiNIA samples are closely related to each other, we were able to use these genotypes to impute the genomes of many close relatives who were genotyped with Affymetrix 10K SNP arrays. Typically, our SardiNIA GWAS analysis thus include approximately 4300 genotyped or imputed individuals. To learn more about the approach see Chen and Abecasis (2007) and Scuteri et al (2007).

=== Planned Updates ===

Several ongoing experiments are expected to gradually improve our GWAS data. First, we expect to integrate Affymetrix 1M SNP chip genotypes into our analyses. Second, we plan to genotype all sampled individuals with the Metabochip, which includes 200,000 SNPs. This will enable us to fill in genotypes for relatives of those genotyped with denser arrays more accurately, because we will more precisely identify shared stretches of chromosome.

== Medical Sequencing Project ==

We are sequencing the genomes of 1,000 individuals to learn about the genetics of blood lipid levels and personality.

== Status as of June 2016 ==

=== Locations of Files for Current Data Freeze of 3839 Samples===

NOTICE: We identified late in the process that two of the samples (22855 and 22385) were actually the same individual. They both should be the same individual 22855. Therefore, there are 3840 sample IDs in each of the files below, but only 22855 should move on to later processes. In future data freezes with this data, these two sequencing sets should be merged into a single 22855 individual.

* List of '''Sample Numbers'''
** The following file contains three columns: [SampleID used in these analyses] [ID supplied by CSCT or Sardinia or other project] [Sequencing core ID (if different)]:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330

* List of paths to '''BAMs''' used in this data freeze (Index file)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_index_20150504.index
** 401 of these samples have some new BAM contribution since the previous data freeze... their BAMs can be found here: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/newsamples.index

* '''Pedigree''' (Not too helpful -- used for SNPCall)
** All Samples: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_ped_20150510.ped

* '''QC''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCStats.txt
** For a list of paths to all of the separate QPLOT files for each sample, see the file: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/GeneratingQCDistributions/QCFileListFinal.txt

* '''SNPCall''' Results (Produced with Gotcloud SNPCall and phased using Beagle4)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''IndelCall''' Results (Produced with Gotcloud Indel)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/indel/final/all.genotypes.sites.vcf.gz
** SNPEff and VEP declarations of Indel types can be found:
*** SNPEff: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/snpEff/*
*** VEP: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/VEP/*
** We used an '''Indel filtering strategy''' composed of many levels.
**# AC must be 1 or greater -- eliminate Indels with AC=0
**# the Indel should overlap with a VNTR region or overlap with another Indel. We used Adrian's annotate indels program to identify such overlaps. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/All.annotated.sites.vcf.gz
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/Overlaps.txt
**# at least 50% of the Indels should have informative AD field (we define "informative" to mean that the sample has U/(R+A+U)<0.50). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter2_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# at least 50% of the Indels should have informative PL field (we define "informative" to mean that the PL field for the sample is anything BUT ././. or 0/0/0). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter1_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# the Indel needs to have BF_LRE_LUD (a Bayes factor comparing a. related & HWE to b. unrelated & HWD) > -10. Higher BF_LRE_LUD should indicate a better Indel. We used Hyun's MiLK program to obtain BF_LRE_LUD values. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/all.genotypes.milk.sites.vcf
*** Overall results from all of the below filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/FinalIndelFilteringStatistics.txt
*** VCFs of Indels after filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.*.PASS.vcf.gz
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.PASS.vcf.gz

* Merged Indel VCF with beagles SNP VCF, then sorted to get the following VCFS:
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/chr*.PASS.beagled.vcf.gz

* Indel and SNP VCFs that have been combined AND BEAGLED AGAIN TOGETHER using Beagle4. '''These are the latest VCFS''':
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/beagle4/beagle4_chr*/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''mtDNA Copy Number''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/sardiniaCopyNumber_include.txt
** By sample copy number results by sample can be found: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/*.CopyNumber.noRand.500000.*.txt

* '''Filtering Samples''': Discrepancy Figures
** We compared chip data from previous work to the sequencing data produced now in hopes of identifying which samples have reliable sequencing data. This comparison was done for each chromosome separately and then combined into an overall discrepancy figure. Only 3189 of the 3840 samples had
** By chromosome discrepancy figures can be found:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/*.diff.discordance_matrix
** Overall discrepancy can be found:
*** '''''IN PROCESS'''''

=== What is Complete ===

=== Future Directions ===

== Key References ==

If you are looking to learn about the project, I strongly recommend that you read the following papers

* Pilia G, Chen WM, Scuteri A, Orru M, Albai G, Dei M, Lai S, Usala G, Lai M, Loi P, Mameli C, Vacca L, Deiana M, Olla N, Masala M, Cao A, Najjar SS, Terracciano A, Nedorezov T, Sharov A, Zonderman AB, Abecasis GR, Costa P, Lakatta E and Schlessinger D (2006). Heritability of Cardiovascular and Personality Traits in 6,148 Sardinians. PLoS Genet 2:1207-1223 [[http://www.sph.umich.edu/csg/abecasis/publications/16934002.html Abstract and PDF]]

* Scuteri A, Sanna S, Chen WM, Uda M, Albai G, Strait J, Najjar S, Nagarajah R, Orru M, Usala G, Dei M, Lai S, Maschio A, Busonero F, Mulas A, Ehret GB, Fink AA, Weder A, Cooper R, Galan P, Chakravarti A, Schlessinger D, Cao A, Lakatta E and Abecasis GR (2007). Genome Wide Association Scan shows Genetic Variants in the FTO gene are Associated with Obesity Related Traits PLoS Genetics 3:1200-10 [[http://www.sph.umich.edu/csg/abecasis/publications/PLOS-Obesity-Scan.html Abstract and PDF]]

* Chen WM and Abecasis GR (2007). Family-based association tests for genomewide association scans. Am J Hum Genet 81:913-26 [[http://www.sph.umich.edu/csg/abecasis/publications/17924335.html Abstract and PDF]]

SardiNIA

2016-06-27T16:05:40Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

== Project Leaders ==

* David Schlessinger (National Institutes on Aging, Baltimore)
* Manuela Uda (National Research Center, Cagliari, Italy)
* Goncalo Abecasis (University of Michigan, Ann Arbor)

== Genomewide Association Study ==

We carried out an initial genomewide association by genotyping 1405 samples with Affymetrix 500K SNP arrays. Then, because SardiNIA samples are closely related to each other, we were able to use these genotypes to impute the genomes of many close relatives who were genotyped with Affymetrix 10K SNP arrays. Typically, our SardiNIA GWAS analysis thus include approximately 4300 genotyped or imputed individuals. To learn more about the approach see Chen and Abecasis (2007) and Scuteri et al (2007).

=== Planned Updates ===

Several ongoing experiments are expected to gradually improve our GWAS data. First, we expect to integrate Affymetrix 1M SNP chip genotypes into our analyses. Second, we plan to genotype all sampled individuals with the Metabochip, which includes 200,000 SNPs. This will enable us to fill in genotypes for relatives of those genotyped with denser arrays more accurately, because we will more precisely identify shared stretches of chromosome.

== Medical Sequencing Project ==

We are sequencing the genomes of 1,000 individuals to learn about the genetics of blood lipid levels and personality.

== Status as of June 2016 ==

=== Locations of Files for Current Data Freeze of 3839 Samples===

NOTICE: We identified late in the process that two of the samples (22855 and 22385) were actually the same individual. They both should be the same individual 22855. Therefore, there are 3840 sample IDs in each of the files below, but only 22855 should move on to later processes. In future data freezes with this data, these two sequencing sets should be merged into a single 22855 individual.

* List of '''Sample Numbers'''
**

* List of paths to '''BAMs''' used in this data freeze (Index file)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_index_20150504.index
** 401 of these samples have some new BAM contribution since the previous data freeze... their BAMs can be found here: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/newsamples.index

* '''Pedigree''' (Not too helpful -- used for SNPCall)
** All Samples: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/FINAL_ped_20150510.ped

* '''QC''' Summary
**

* '''SNPCall''' Results (Produced with Gotcloud SNPCall and phased using Beagle4)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPCALL/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''IndelCall''' Results (Produced with Gotcloud Indel)
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/indel/final/all.genotypes.sites.vcf.gz
** SNPEff and VEP declarations of Indel types can be found:
*** SNPEff: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/snpEff/*
*** VEP: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/VEP/*
** We used an '''Indel filtering strategy''' composed of many levels.
**# AC must be 1 or greater -- eliminate Indels with AC=0
**# the Indel should overlap with a VNTR region or overlap with another Indel. We used Adrian's annotate indels program to identify such overlaps. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/All.annotated.sites.vcf.gz
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/AnnotateIndels/Overlaps.txt
**# at least 50% of the Indels should have informative AD field (we define "informative" to mean that the sample has U/(R+A+U)<0.50). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter2_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# at least 50% of the Indels should have informative PL field (we define "informative" to mean that the PL field for the sample is anything BUT ././. or 0/0/0). Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/filter1_output.txt
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/INDEL_filtering_final.txt
**# the Indel needs to have BF_LRE_LUD (a Bayes factor comparing a. related & HWE to b. unrelated & HWD) > -10. Higher BF_LRE_LUD should indicate a better Indel. We used Hyun's MiLK program to obtain BF_LRE_LUD values. Results for Indels on all chromosomes can be found here:
**#* /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/MiLK_filtering/all.genotypes.milk.sites.vcf
*** Overall results from all of the below filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/FinalIndelFilteringStatistics.txt
*** VCFs of Indels after filtering can be found here:
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.*.PASS.vcf.gz
**** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/filterIndels/FINAL/all.genotypes.PASS.vcf.gz

* Merged Indel VCF with beagles SNP VCF, then sorted to get the following VCFS:
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/chr*.PASS.beagled.vcf.gz

* Indel and SNP VCFs that have been combined AND BEAGLED AGAIN TOGETHER using Beagle4. '''These are the latest VCFS''':
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/SNPINDEL/beagle4/beagle4_chr*/beagle4/chr*/chr*.filtered.PASS.beagled.vcf.gz

* '''mtDNA Copy Number''' Summary
** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/sardiniaCopyNumber_include.txt
** By sample copy number results by sample can be found: /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/MTDNA_COPYNUMBER/Coverage_IncludingUnpairedReads/*.CopyNumber.noRand.500000.*.txt

* '''Filtering Samples''': Discrepancy Figures
** We compared chip data from previous work to the sequencing data produced now in hopes of identifying which samples have reliable sequencing data. This comparison was done for each chromosome separately and then combined into an overall discrepancy figure. Only 3189 of the 3840 samples had
** By chromosome discrepancy figures can be found:
*** /net/sardinia/progenia/SardiNIA/VariantCalling_20150330/filteringSamples/chr*/*.diff.discordance_matrix
** Overall discrepancy can be found:
*** '''''IN PROCESS'''''

* '''Sample Name Conversion'''
** The CSCT samples had two different names.... a numeric and an alphanumeric. The conversion key can be found here:
***/net/sardinia/progenia/SardiNIA/VariantCalling_20150330/INDELCALL/SampleNameChanges.txt
** The Sardinia samples had a Sardinia ID and a Sequencing Core ID. The conversion key can be found here:
*** '''''IN PROCESS'''''

=== What is Complete ===

=== Future Directions ===

== Key References ==

If you are looking to learn about the project, I strongly recommend that you read the following papers

* Pilia G, Chen WM, Scuteri A, Orru M, Albai G, Dei M, Lai S, Usala G, Lai M, Loi P, Mameli C, Vacca L, Deiana M, Olla N, Masala M, Cao A, Najjar SS, Terracciano A, Nedorezov T, Sharov A, Zonderman AB, Abecasis GR, Costa P, Lakatta E and Schlessinger D (2006). Heritability of Cardiovascular and Personality Traits in 6,148 Sardinians. PLoS Genet 2:1207-1223 [[http://www.sph.umich.edu/csg/abecasis/publications/16934002.html Abstract and PDF]]

* Scuteri A, Sanna S, Chen WM, Uda M, Albai G, Strait J, Najjar S, Nagarajah R, Orru M, Usala G, Dei M, Lai S, Maschio A, Busonero F, Mulas A, Ehret GB, Fink AA, Weder A, Cooper R, Galan P, Chakravarti A, Schlessinger D, Cao A, Lakatta E and Abecasis GR (2007). Genome Wide Association Scan shows Genetic Variants in the FTO gene are Associated with Obesity Related Traits PLoS Genetics 3:1200-10 [[http://www.sph.umich.edu/csg/abecasis/publications/PLOS-Obesity-Scan.html Abstract and PDF]]

* Chen WM and Abecasis GR (2007). Family-based association tests for genomewide association scans. Am J Hum Genet 81:913-26 [[http://www.sph.umich.edu/csg/abecasis/publications/17924335.html Abstract and PDF]]

SardiNIA

2016-06-27T15:47:02Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

SardiNIA

2016-06-27T15:46:48Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

SardiNIA

2016-06-27T15:46:18Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

SardiNIA

2016-06-27T15:45:05Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

SardiNIA

2016-06-27T15:44:05Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

SardiNIA

2016-06-27T15:41:46Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

SardiNIA

2016-06-27T15:41:03Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

SardiNIA

2016-06-27T15:32:53Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

SardiNIA

2016-06-27T15:32:36Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

SardiNIA

2016-06-27T15:31:28Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

SardiNIA

2016-06-27T15:31:00Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

SardiNIA

2016-06-27T15:29:29Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

SardiNIA

2016-06-27T15:29:07Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

SardiNIA

2016-06-27T15:28:45Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

SardiNIA

2016-06-27T15:00:45Z

Kleckner: /* Status as of June 2016 */

SardiNIA

2016-06-27T15:00:15Z

Kleckner: /* Status as of June 2016 */

SardiNIA

2016-06-27T14:59:43Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

SardiNIA

2016-06-27T14:58:31Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

SardiNIA

2016-06-27T14:57:18Z

Kleckner: /* Locations of Files for Current Data Freeze of 3839 Samples */

SardiNIA

2016-06-27T14:24:23Z

Kleckner: /* Status as of June 2016 */

SardiNIA

2016-06-27T14:04:52Z

Kleckner: /* Locations of Files for Current Data Freeze of 3840 Samples */

SardiNIA

2016-06-27T13:35:46Z

Kleckner: /* Status as of June 2016 */

SardiNIA

2016-06-27T13:31:33Z

Kleckner: /* Status as of June 2016 */

SardiNIA

2016-06-27T13:25:43Z

Kleckner: /* Status as of June 2016 */

SardiNIA

2016-06-27T13:25:22Z

Kleckner: /* Status as of June 2016 */

SardiNIA

2016-06-27T13:19:39Z

Kleckner:

SardiNIA

2016-06-27T13:19:06Z

Kleckner:

SeqShop: Variant Calling and Filtering for SNPs Practical, December 2014

2015-03-30T23:12:57Z

Kleckner: /* Sequnce Alignment Files: BAM Files */

==Introduction==
Main Workshop wiki page: [[SeqShop: December 2014]]

See the [[Media:Dec2014 SeqShop - GotCloud snpcall.pdf|introductory slides]] for an intro to this tutorial.

== Goals of This Session ==
* What we want to learn
** How to generate filtered variant calls for SNPs from BAMs
** Basic variant call file format (VCF)
** How to examine the variants at particular genomic positions
** How to evaluate the quality of SNP calls

== Setup in person at the SeqShop Workshop ==
''This section is specifically for the SeqShop Workshop computers.''
<div class="mw-collapsible mw-collapsed" style="width:600px">
''If you are not running during the SeqShop Workshop, please skip this section.''
<div class="mw-collapsible-content">

{{SeqShopLogin}}

=== Setup your run environment===
This is the same setup you did for the previous tutorial, but you need to redo it each time you log in.

This will setup some environment variables to point you to
* [[GotCloud]] program
* Tutorial input files
* Setup an output directory
** It will leave your output directory from the previous tutorial in tact.
source /net/seqshop-server/home/mktrost/seqshop/setup.txt
* You won't see any output after running <code>source</code>
** It silently sets up your environment
** If you want to view the detail of the setup, type
less /net/seqshop-server/home/mktrost/seqshop/setup.txt
and press 'q' to finish.

<div class="mw-collapsible mw-collapsed" style="width:200px">
View setup.txt
<div class="mw-collapsible-content">
[[File:setup.png|500px]]
</div>
</div>
</div>
</div>

== Setup when running on your own outside of the SeqShop Workshop ==
''This section is specifically for running on your own outside of the SeqShop Workshop.''
<div class="mw-collapsible" style="width:600px">
''If you are running during the SeqShop Workshop, please skip this section.''
<div class="mw-collapsible-content">

This tutorial builds on the alignment tutorial, if you have not already, please first run that tutorial: [[SeqShop:_Sequence_Mapping_and_Assembly_Practical|Alignment Tutorial]]

{{SeqShopRemoteEnv}}
</div>
</div>

== Examining GotCloud SnpCall Input files ==
=== Sequence Alignment Files: BAM Files ===
Per sample BAM files contain sequence reads that are mapped to positions in the genome.

For a reminder on how to look at/read BAM files, see: [[SeqShop:_Sequence_Mapping_and_Assembly_Practical#BAM_Files|SeqShop Aligment: BAM Files]]

For this tutorial, we will use the 4 BAMs produced in the [[SeqShop: Sequence Mapping and Assembly Practical]] as well as with 58 BAMs that were pre-aligned to that 1MB region of chromosome 22.

=== Reference Files ===
Reference files can be downloaded with GotCloud or from other sources.
* For this practical, I already downloaded them for you.
* See [[GotCloud: Genetic Reference and Resource Files]] for more information on downloading/generating reference files

For GotCloud snpcall, you need:
# Reference genome FASTA file
#* Contains the reference base for each position of each chromosome
#** Used to compare bases in sequence reads to the reference positions they mapped to
#** Used to identify SNPs/variations in the sequence reads
#* Additional information on the FASTA format: http://en.wikipedia.org/wiki/FASTA_format
# VCF (variant call format) files with chromosomes/positions
#* indel - contains known insertions & deletions to help with filtering
#* omni - used as likely true positives for SVM filtering
#* hapmap - used as likely true positives for SVM filtering and for generating summary statistics
#* dbsnp - used for generating summary statistics

We looked at them yesterday, but you can take another look at the chromosome 22 reference files included for this tutorial:
ls ${SS}/ref22

<ul>
<div class="mw-collapsible mw-collapsed" style="width:200px">
<li>View Screenshot</li>
<div class="mw-collapsible-content">
[[File:RefDir.png|700px]]
</div>
</div>
</ul>

=== GotCloud BAM List File ===
The [[GotCloud:_Variant_Calling_Pipeline#BAM_List_File|BAM list file]] points GotCloud to the BAM files
* generated by the alignment pipeline

Look at the BAM list file the alignment pipeline generated
cat ${OUT}/bam.list

;What is the path to the BAM file for sample HG00640?
<ul>
<div class="mw-collapsible mw-collapsed" style="width:200px">
<li>Answer:</li>
<div class="mw-collapsible-content">
<ul>
<li>/net/seqshop-server/home/YourUserName/out/bams/HG00640.recal.bam</li>
[[File:BamindexNew.png|500px]]
</div>
</div>
</ul>
</ul>

The alignment pipeline only processed 4 samples, but for snpcall, we want to run on 62 samples.
* The other 58 samples were already aligned:
ls ${SS}/bams

Look at the BAM list file for those BAMs:
less ${SS}/bams/bam.list

Remember, use <code>'q'</code> to exit out of <code>less</code>
q

;Do you notice a difference between this list and yours?
<ul>
<div class="mw-collapsible mw-collapsed" style="width:550px">
<li>Answer:</li>
<div class="mw-collapsible-content">
<ul>
<li>It doesn't have a full path to the BAM file, while your list has /home/...</li>
[[File:BamList1.png|300px]]
<li>That's ok, we will use the <code>--base_prefix ${SS}</code> command-line option to prefix the BAM paths</li>
<li>Alternatively, we could have set BAM_PREFIX in <code>gotcloud.conf</code> to the path to the BAMs
<pre>BAM_PREFIX = /net/seqshop-server/home/mktrost/seqshop/example</pre> </li>
<ul>
<li>NOTE: the conf file can't interpret ${SS} environment variables or '~', so you would have to specify the full path</li>
<li>We just used the command-line option for this tutorial since this path will vary by user when running outside the workshop.</li>
</ul>
</div>
</div>
</ul>
</ul>

We need to add these BAMs to our list
* Append the bam.list from the pre-aligned BAMs to the one you generated from the alignment pipeline
** '''Be sure to do this command just once'''
cat ${SS}/bams/bam.list >> ${OUT}/bam.list
* ">>" will append to the file that follows it
** Check that your BAM list is the correct size
**:<pre>wc -l ${OUT}/bam.list</pre>
*** <code>wc -l</code> counts the number of lines in the file
*** Should be 62

Verify your BAM list contains the additional BAMs
less ${OUT}/bam.list

Remember, use <code>'q'</code> to exit out of <code>less</code>
q

;Do you see both sets of BAMs?
<ul>
<div class="mw-collapsible mw-collapsed" style="width:500px">
<li>Annotated Screenshot:</li>
<div class="mw-collapsible-content">
<ul>
<li>If not, let me know</li>
[[File:BamList2.png|400px]]
</div>
</div>
</ul>
</ul>

=== GotCloud Configuration File ===
We will use the same configuration file as we used yesterday in GotCloud Align.

See [[SeqShop:_Sequence_Mapping_and_Assembly_Practical#GotCloud Configuration File|SeqShop: Alignment: GotCloud Configuration File]] for more details
* Note we want to limit snpcall to just chr22 so the configuration already has <code>CHRS = 22</code> (default was 1-22 & X).

For more information on configuration, see: [[GotCloud:_Variant_Calling_Pipeline#Configuration_File|GotCloud snpcall: Configuration File]]
* Contains information on how to configure for exome/targeted sequencing

== Run GotCloud SnpCall ==
[[File:SnpcallDiagramNew.png|500px]]

Now that we have all of our input files, we need just a simple command to run:
* When running at home if you don't have 6 CPUs, reduce the --numjobs setting (it will take longer to run).
${GC}/gotcloud snpcall --conf ${SS}/gotcloud.conf --numjobs 6 --region 22:36000000-37000000 --base_prefix ${SS} --outdir ${OUT}
* <code>${GC}/gotcloud</code> runs GotCloud
* <code>snpcall</code> tells GotCloud you want to run the snpcall pipeline.
* <code>--conf</code> tells GotCloud the name of the configuration file to use.
** The configuration for this test was downloaded with the seqshop input files.
* --numjobs tells GotCloud how many jobs to run in parallel
** Depends on your system
* --region 22:36000000-37000000
** The sample files are just a small region of chromosome 22, so to save time, we tell GotCloud to ignore the other regions
* <code>--base_prefix</code> tells GotCloud the prefix to append to relative paths.
** The Configuration file cannot read environment variables, so we need to tell GotCloud the path to the input files, ${SS}
** Alternatively, gotcloud.conf could be updated to specify the full paths
* <code>--outdir</code> tells GotCloud where to write the output.
** This could be specified in gotcloud.conf, but to allow you to use the ${OUT} to change the output location, it is specified on the command-line

<div class="mw-collapsible mw-collapsed" style="width:500px">
Curious if it started running properly? Check out this screenshot:
<div class="mw-collapsible-content">
[[File:SnpcallStartNew.png|550px]]
</div>
</div>
This should take about 5-8 minutes to run.
* It should end with a line like: <code>Commands finished in 402 secs with no errors reported</code>

If you cancelled GotCloud part way through, just rerun your GotCloud command and it will pick up where it left off.

If you want to understand more detailed step of GotCloud SNP calling, here is a schematic picture with a little bit more details

[[File:Gotcloudoverview.png|600px]]

== Examining GotCloud SnpCall Output ==
Let's look at the output directory:
ls ${OUT}

;Do you see any new files or directories?
<div class="mw-collapsible mw-collapsed" style="width:350px">
* View Annotated Screenshot:
<div class="mw-collapsible-content">
[[File:gcsnpcallOutNew.png|500px]]
</div>
</div>

Let's look at the vcfs directory:
ls ${OUT}/vcfs
Just a <code>chr22</code> directory, so look inside of there:
ls ${OUT}/vcfs/chr22
;Can you identify the final filtered VCF and the associated summary file?
<div class="mw-collapsible mw-collapsed" style="width:350px">
* Answer & annotated directory listing:
<div class="mw-collapsible-content">
<ul>
<li>Filtered VCF (SVM & hard filters): chr22.filtered.vcf.gz</li>
<li>Summary file: chr22.filtered.sites.vcf.summary</li>
</ul>
[[File:vcfsout.png|600px]]
</div>
</div>

Now, let's look in the split directory for the VCF with just the passing variants:
ls ${OUT}/split/chr22
;Which file do you think is the one you want?
<div class="mw-collapsible mw-collapsed" style="width:350px">
* Answer:
<div class="mw-collapsible-content">
<ul>
<li>chr22.filtered.PASS.vcf.gz</li>
</ul>
[[File:splitOut.png|600px]]
</div>
</div>

=== Filtering Summary Statistics ===

cat ${OUT}/vcfs/chr22/chr22.filtered.sites.vcf.summary

<div class="mw-collapsible mw-collapsed" style="width:250px">
View Screenshot
<div class="mw-collapsible-content">
[[File:filterSumNew.png|700px]]
</div>
</div>

'''To understand how to interpret the filtering summary statistics, please refer to [[Understanding vcf-summary output]]'''

=== Filtered VCF ===

Let's look at the filtered sites file.
less -S ${OUT}/vcfs/chr22/chr22.filtered.sites.vcf

* Scroll down until you find some variants.
** Use space bar to jump a full page
** Use down arrow to move down one line
* Scroll right: lots of info fields, but no per sample genotype information

;What is the first filtered out variant that you find & what filter did it fail?
<div class="mw-collapsible mw-collapsed" style="width:250px">
* Answer:
<div class="mw-collapsible-content">
<ul>
<li>It failed SVM filter</li>
</ul>
[[File:SvmFiltNew.png|550px]]
</div>
</div>

Remember, use <code>'q'</code> to exit out of <code>less</code>
q

Now, let's look at the filtered file with genotypes.
zless -S ${OUT}/vcfs/chr22/chr22.filtered.vcf.gz

* Scroll down until you find some variants.
** Use space bar to jump a full page
** Use down arrow to move down one line
* Scroll right until you should see per sample genotype information

Remember, use <code>'q'</code> to exit out of <code>less</code>
q

<div class="mw-collapsible mw-collapsed" style="width:250px">
* View annotated screenshot:
<div class="mw-collapsible-content">
[[File:SvmFiltGLNew.png|550px]]
</div>
</div>

=== Passing SNPs ===

Let's look at the file of just the pass sites:
zless -S ${OUT}/split/chr22/chr22.filtered.PASS.vcf.gz

* Scroll down: they all look like they <code>PASS</code>

Remember, use 'q' to exit out of less
q

Let's check if they are all PASS.
zcat ${OUT}/split/chr22/chr22.filtered.PASS.vcf.gz |grep -v "^#"| cut -f 7| grep -v "PASS"
It will return nothing since there are no non-passing variants in this file.
<div class="mw-collapsible mw-collapsed" style="width:450px">
;Want an explanation of this command?
<div class="mw-collapsible-content">
* zcat ...: uncompress the zipped VCF
* '|' : this takes the output of one command and sends it as input to the next
* grep -v "^#" : exclude any lines that start with "#" - headers
* cut -f 7 : extract the FILTER column (the 7th column)
* grep -v "PASS" : exclude any rows that have a "PASS" in the FILTER column
</div>
</div>

Compare that to the filtered file we looked at before:
zcat ${OUT}/vcfs/chr22/chr22.filtered.vcf.gz |grep -v "^#"| cut -f 7| grep -v "PASS"
;Do you see any filters?
<div class="mw-collapsible mw-collapsed" style="width:450px">
*Answer
<div class="mw-collapsible-content">
* Yes
** It should have scrolled and you should see filters like:
*** INDEL5;SVM
*** INDEL5
*** SVM
</div>
</div>

== GotCloud Genotype Refinement ==
To improve the quality of the genotypes, we run a genotype refinement pipeline.

This pipeline runs [http://faculty.washington.edu/browning/beagle/beagle.html Beagle] & thunder.

=== Genotype Refinement Input ===
The GotCloud genotype refinement pipeline takes as input ${OUT}/split/chr22/chr22.filtered.PASS.vcf.gz (the VCF file of PASS'ing SNPs from snpcall).

The bam list and the configuration file we used for GotCloud snpcall will tell GotCloud genotype refinement everything it needs to know, so no new input files need to be prepared.

Note: the configuration file overrides the THUNDER command to make it go faster than the default settings so the tutorial will run faster:
[[File:thunderConf.png|600px]]

=== Running GotCloud Genotype Refinement ===
Since everything is setup, just run the following command (very similar to snpcall).
${GC}/gotcloud ldrefine --conf ${SS}/gotcloud.conf --numjobs 6 --region 22:36000000-37000000 --base_prefix ${SS} --outdir ${OUT}

* Beagle will take about 1-3 minutes to complete
* Thunder will automatically run and will take another 2-4 minutes

<div class="mw-collapsible mw-collapsed" style="width:350px">
When completed, it should look like this:
<div class="mw-collapsible-content">
[[File:GcldrefineOutNew.png]]
</div>
</div>

=== Genotype Refinement Output ===

; What's new in the output directory?

<ul>
<div class="mw-collapsible mw-collapsed" style="width:500px">
<li>Answer</li>
<div class="mw-collapsible-content">
:<pre>ls ${OUT}</pre>
<ul>
<li><code>beagle</code> directory : Beagle output</li>
<li><code>thunder</code> directory : Thunder output</li>
<li><code>umake.beagle.*</code> : Contain the configuration & steps used in GotCloud beagle</li>
<li><code>umake.thunder.*</code> : Contain the configuration & steps used in GotCloud thunder</li>
</ul>
</div>
</div>
</ul>

Let's take a look at that interesting location we found in the [[SeqShop:_Sequence_Mapping_and_Assembly_Practical#Accessing_BAMs_by_Position|alignment tutorial]] : chromosome 22, positions 36907000-36907100

Use tabix to extract that from the VCFs:
${GC}/bin/tabix ${OUT}/thunder/chr22/ALL/thunder/chr22.filtered.PASS.beagled.ALL.thunder.vcf.gz 22:36907000-36907100 |less -S

Remember, type 'q' to quit less.
q

;Are there any variants in this region?
<ul>
<div class="mw-collapsible mw-collapsed" style="width:500px">
<li>Answer:</li>
<div class="mw-collapsible-content">
<ul>
<li>Yes!</li>
<li>Positions:</li>
<ul>
<li><code>36907001</code>; Ref: T, Alt: C - that's what we saw before</li>
<li><code>36907098</code>; Ref: T, Alt: C - that's what we saw before</li>
</ul>
</ul>
</div>
</div>
</ul>

;What is HG00551's genotype at these positions?
#First check which sample number HG00551 is:
zcat ${OUT}/thunder/chr22/ALL/thunder/chr22.filtered.PASS.beagled.ALL.thunder.vcf.gz |grep "#CHROM"
* That will help you figure out it's genotype.
* Rerun the tabix command and scroll to find HG00551's genotype:
${GC}/bin/tabix ${OUT}/thunder/chr22/ALL/thunder/chr22.filtered.PASS.beagled.ALL.thunder.vcf.gz 22:36907000-36907100 |less -S

<ul>
<div class="mw-collapsible mw-collapsed" style="width:500px">
<li>Answer:</li>
<div class="mw-collapsible-content">
<ul>
<li>It is the first sample</li>
<li><code>0|1</code>: Heterozygous (although low GQ - quality)</li>
<li><code>1|1</code>; Homozygous Alt (C)</li>
</ul>
</div>
</div>
</ul>

Remember, type 'q' to quit less.
q

=== Did I find interesting variants? ===

The region we selected contains ''APOL1'' gene, which is known to play an important role in kidney diseases such as nephrotic syndrome. One of the non-synonymous risk allele, <code>rs73885139</code> located at position <code>22:36661906</code> increases the risk of nephrotic syndrome by >2-folds. Let's see if we found the interesting variant by looking at the VCF file by position.

${GC}/bin/tabix ${OUT}/vcfs/chr22/chr22.filtered.vcf.gz 22:36661906 | head -1

Did you see a variant at the position?

22 36661906 . A G 23 PASS DP=409;MQ=59;NS=62;AN=124;AC=2;AF=0.013847;
AB=0.4169;AZ=-0.3525;FIC=-0.0089;SLRT=-0.0071;LBS=36,36,0,0,1,1,0,0;OBS=145,191,0,0,3,2,0,0;STR=-0.040;
STZ=-0.740;CBR=0.008;CBZ=0.144;IOR=0.000;IOZ=-1.370;AOI=-5.614;AOZ=-4.243;LQR=0.178;MQ0=0.000;
MQ10=0.000;MQ20=0.000;MQ30=0.000;SVM=1.45191 GT:DP:GQ:PL 0/0:4:28:0,12,65

Let's check the sequence data to confirm that the variant really exists

${GC}/bin/samtools tview ${SS}/bams/HG01242.recal.bam ${SS}/ref22/human.g1k.v37.chr22.fa

* Type 'g' to go to a specific position
* Type 22:36661906 to move to the position
* Press arrows to move between positions
* Press 'b' if you want to color by base quality
* Press '?' for more help

<div class="mw-collapsible mw-collapsed" style="width:250px">
View Screenshot
<div class="mw-collapsible-content">
[[File:SamtoolstviewsnpNew.png|600px]]
</div>
</div>

=== Improvements ===

Let's get some information on the BEAGLE VCF:

perl ${GC}/scripts/bed-diff.pl --vcf1 ${SS}/ref22/1kg.omni.chr22.36Mb.vcf.gz --vcf2 ${OUT}/beagle/chr22/chr22.filtered.PASS.beagled.ALL.vcf.gz --out ${OUT}/diffs/bedDiff.beagle

Look at the results:
more ${OUT}/diffs/bedDiff.beagle.summary

<div class="mw-collapsible mw-collapsed" style="width:400px">
*Results
<div class="mw-collapsible-content">
OVERALL: 43588 44293 0.9841
NREF-EITHER: 19644 20349 0.9654
NMAJ-EITHER: 14560 15265 0.9538

HOMREF: 23944 91 0 0.9962
HET: 355 11936 172 0.9577
HOMALT: 6 81 7708 0.9888

HOMMAJ: 29028 112 4 0.9960
HET: 380 11936 147 0.9577
HOMMIN: 2 60 2624 0.9769
</div>
</div>

Now, let's see if it improved after running Thunder VCF:
perl ${GC}/scripts/bed-diff.pl --vcf1 ${SS}/ref22/1kg.omni.chr22.36Mb.vcf.gz --vcf2 ${OUT}/thunder/chr22/ALL/thunder/chr22.filtered.PASS.beagled.ALL.thunder.vcf.gz --out ${OUT}/diffs/bedDiff.thunder

Look at the results:
more ${OUT}/diffs/bedDiff.thunder.summary

<div class="mw-collapsible mw-collapsed" style="width:400px">
*Results
<div class="mw-collapsible-content">
OVERALL: 43711 44293 0.9869
NREF-EITHER: 19777 20359 0.9714
NMAJ-EITHER: 14715 15297 0.9620

HOMREF: 23934 101 0 0.9958
HET: 272 12084 107 0.9696
HOMALT: 5 97 7693 0.9869

HOMMAJ: 28996 145 3 0.9949
HET: 272 12084 107 0.9696
HOMMIN: 2 53 2631 0.9795
</div>
</div>

There is an improvement.

== What is GotCloud snpcall doing? ==
To run GotCloud, you really just needed a single command.

Well, that one command runs many steps. Here is a diagram of all the steps.

[[File:SnpcallPipeline.jpg|1200px]]

Aren't you glad you didn't have to configure & run each one yourself?

== Return to Workshop Wiki Page ==
Return to main workshop wiki page: [[SeqShop: December 2014]]

GotCloud: Variant Calling Pipeline

2015-03-30T22:54:59Z

Kleckner: /* Running the Automatic Test */

Back to parent: [[GotCloud]]

The Variant Calling Pipeline (previously called 'UMAKE') makes genotype calls from recalibrated BAM files. These genotype calls are output into [http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 VCF (Variant Call Format) files].

== Running the GotCloud Variant Calling Pipeline ==

The variant calling pipeline (umake) is run using <code>gotcloud snpcall</code> and <code>gotcloud ldrefine</code>.

===Running the Automatic Test===

The automatic test runs the variant calling pipeline on a small test set and checks the results against expected results validating that GotCloud is installed correctly.

*Run <code>snpcall</code> pipeline test:
gotcloud snpcall --test OUTPUT_DIR
** Where OUTPUT_DIR is the directory where you want to store the test results
** If you see <code>Successfully ran the test case, congratulations!</code>, then you are ready to run snpcall on your own samples.
*Run <code>ldrefine</code> pipeline test:
gotcloud ldrefine --test OUTPUT_DIR
** Where <code>OUTPUT_DIR</code> is the directory where you want to store the test results
** If you see <code>Successfully ran the test case, congratulations!</code>, then you are ready to run ldrefine on your own samples.

== Overview of Variant Calling Pipeline Steps ==
Here is an overview of the Variant Calling Pipeline:

[[File: umakeSteps.png]]

For more information on the filters applied during the Variant Calling Pipeline, see, [[GotCloud: Filters]].

== Input Data==
* [[#BAM Files|Aligned/Processed/Recalibrated BAM files]]
* [[#BAM List File|BAM list file containing Sample IDs & BAM file names]]
* [[#Reference Files|Reference files]]
* (Optional) [[#Configuration File|Configuration file to override default options]]

=== BAM Files ===
The BAM files need to be duplicate-marked and base-quality recalibrated in order to obtain high quality SNP calls. Generating these BAM files from original FASTQs is automatically done as part of the [[Alignment Pipeline]] of GotCloud.

=== BAM List File ===
* Automatically created when running the GotCloud [[Alignment Pipeline]]
* Each line of the BAM list file represents a single individual

Columns:
# sample id
# comma separated population labels (optional column)
# BAM File 1 (preferable to have full paths to BAM files)
# BAM File 2 (if more than 1 BAM per sample)
:...

: # BAM File N (if more than 1 BAM per sample)
[SAMPLE_ID] [COMMA SEPARATED POPULATION LABELS] [BAM_FILE1] [BAM_FILE2] ...
or
[SAMPLE_ID] [BAM_FILE1] [BAM_FILE2] ...

* Notes:
** tab delimited
** multiple BAMs per individual may be provided, but should all be on the same line of the list file
** population label is optional - it will default to <code>ALL</code>
*** only used by Thunder (part of ldrefine pipeline)
*** if all samples are from the same population, population label can be skipped or you can just specify <code>ALL</code> for the population label for each sample.

The path to the BAM List file is defaulted to the <code>outputDirectory/bam.list</code>. It can be overridden by setting <code>--bamlist</code>, <code>--bam_list</code>, or <code>--list</code> on the command-line or by setting BAM_LIST in your configuration file to the path to the BAM List File. See [[#Required_Options|Required Options]] for more information.

=== Reference Files ===
See [[GotCloud: Genetic Reference and Resource Files]] for detailed information about the multiple required reference files for the variant calling pipeline, including:
* How to obtain default references
* Configuration keys & default values
* How to generate your own references
* How to point GotCloud to your reference files

Required Reference File Types:
* [[GotCloud: Genetic Reference and Resource Files#Reference fasta Files|Reference fasta Files]]
* [[GotCloud: Genetic Reference and Resource Files#DBSNP VCF Files|DBSNP VCF Files]]
* [[GotCloud: Genetic Reference and Resource Files#HapMap3 VCF Files|HapMap3 VCF Files]]
* [[GotCloud: Genetic Reference and Resource Files#OMNI VCF Files|OMNI VCF Files]]
* [[GotCloud: Genetic Reference and Resource Files#INDEL VCF File(s)|INDEL VCF File(s)]]

=== Configuration File ===
{{:GotCloud: Configuration}}

See [[#Variant Calling Command-line Options/Configuration Settings|Variant Calling Command-line Options/Configuration Settings]] for more information on Configuration options.

==== Example Configuration File ====
Example configuration file where reference files happen to be stored in /path/reference, and bam index file in path/freeze5
CHRS = 20 22
BAM_LIST = /path/freeze5.bam.list
OUT_DIR = /path/freeze5/output
REF_DIR = /path/reference/
REF = $(REF_DIR)/hs37d5.fa
INDEL_PREFIX = $(REF_DIR)/1kg.pilot_release.merged.indels.sites.hg19
HM3_VCF = $(REF_DIR)/hapmap3_r3_b37.sites.vcf.gz
DBSNP_VCF = $(REF_DIR)/dbsnp_135.b37.sites.vcf.gz

== Variant Calling Command-line Options/Configuration Settings ==
{{:GotCloud: Variant Calling Options}}

== Use Cases & Recommended Settings ==
=== Single Sample Processing ===
To run single sample processing we recommend adding the following settings to your configuration file:
UNIT_CHUNK = 20000000
MODEL_GLFSINGLE = TRUE
MODEL_SKIP_DISCOVER = FALSE
MODEL_AF_PRIOR = TRUE
VCF_EXTRACT = $(REF_DIR)/snpOnly.vcf.gz
EXT = $(REF_DIR)/ALL.chrCHR.phase3.combined.sites.unfiltered.vcf.gz $(REF_DIR)/chrCHR.filtered.sites.vcf.gz

Explanation of these settings:
* <code>UNIT_CHUNK</code> - since this is only 1 sample, process larger regions at a time than default
* <code>MODEL_GLFSINGLE</code> - single sample, so model glfsingle
* <code>MODEL_SKIP_DISCOVER</code> - do not skip the variant discovery step
* <code>MODEL_AF_PRIOR</code> - use AF prior for genotyping
* <code>VCF_EXTRACT</code> - VCF file to use for extracting the site information to genotype
** This file is included in the latest reference release: [[GotCloud:_Genetic_Reference_and_Resource_Files#hs37d5-db142|hs37d5-db142]]
* <code>EXT</code> - VCF reference files to use for the external filtering
** These files are included in the latest reference release: [[GotCloud:_Genetic_Reference_and_Resource_Files#hs37d5-db142|hs37d5-db142]]

== Running ==

Running variant calling is straightforward:

<code>
'''gotcloud snpcall --conf vc.conf --numjobs 2
'''gotcloud ldrefine --conf vc.conf --numjobs 2
</code>

* Replace <code>vc.conf</code> with the path/name of the user's configuration file
** If you are not overriding any defaults, you can alternatively specify <code>--list path/bam.list</code> replacing <code>path/bam.list</code> with the path/name of your BAM list file.
* Replace <code>2</code> following <code>--numjobs</code> with the number of jobs to be run in parallel
* If <code>OUT_DIR</code> is not defined in the configuration file, add <code>--outdir</code> followed by the path to the user's desired output directory.

=== Running on a Cluster ===
See [[#Cluster Configuration|Cluster Configuration]] for information on how to configure GotCloud to run on a cluster.

== Results ==

If there is a failure, you should see a message like:
make: *** [...] Error 1
Where ... is filled in with other text indicating what step failed.

On SNP Call success, you should see the following output sub-directories under your output directory:
* glfs with a bams & samples subdirectory
* pvcfs with a subdirectory per chromosome and then per region
* '''split''' with a subdirectory per chromosome
* '''vcfs''' with a subdirectory per chromosome
* (optionally your target directory)

Under the '''vcf/chrXX''' directory, there should be:
* chrXX.filtered.sites.vcf
* chrXX.filtered.sites.vcf.norm.log
* chrXX.filtered.sites.vcf.summary
* '''chrXX.filtered.vcf.gz''' - final filtered variant call file
* chrXX.filtered.vcf.gz.OK
* chrXX.filtered.vcf.gz.tbi
* chrXX.hardfiltered.sites.vcf
* chrXX.hardfiltered.sites.vcf.log
* chrXX.hardfiltered.sites.vcf.summary
* chrXX.hardfiltered.vcf.gz
* chrXX.hardfiltered.vcf.gz.OK
* chrXX.hardfiltered.vcf.gz.tbi
* chrXX.merged.sites.vcf
* chrXX.merged.stats.vcf
* chrXX.merged.vcf
* chrXX.merged.vcf.OK

The .merged.vcf is the merged together versions of the separate regions in the same chromosome.

The filtered is the merged.vcf after it has been run through filters and is marked with PASS/FAIL.

Under the '''split/chrXX''' directory, there should be:
* chrXX.filtered.PASS.split.[N].vcf.gz
* chrXX.filtered.PASS.split.err
* chrXX.filtered.PASS.split.vcflist
* '''chrXX.filtered.PASS.gz''' - final variant call file with only PASS variants
* subset.OK

VerifyBamID

2015-03-20T01:25:08Z

Kleckner: /* Column information in the output files */

[[Category:Software|VerifyBamID]]

'''verifyBamID''' is a software that verifies whether the reads in particular file match previously known genotypes for an individual (or group of individuals), and checks whether the reads are contaminated as a mixture of two samples. '''verifyBamID''' can detect sample contamination and swaps when external genotypes are available. When external genotypes are not available, '''verifyBamID''' still robustly detects sample swaps.

== Download verifyBamID ==

To get a copy of verifyBamId, go to: https://github.com/statgen/verifyBamID/releases

Select the latest release and download in one of 3 ways:
# Binary expected to run in Ubuntu x64 platform. In other platforms, please download the source distribution and build it.
#* verifyBamID.#.#.#.gz
#* You will need to run "gunzip" on the .gz file
# Souce Code including libStatGen (uses a fixed version of libStatGen)
#* verifyBamIDLibStatGen.#.#.#.tgz
#* Run "tar xvf" on this file. Cd into the resulting directory & type make.
# Source Code without libStatGen (allows alternative/newer versions of libStatGen)
#* Source code (tar.gz) or Source code (zip)
#* You will need to download libStatGen separately if you do not already have it.

To get a copy of older releases go to the [http://www.sph.umich.edu/csg/kang/verifyBamID/download VerifyBamID Download] download page.

== Join in verifyBamID mailing list ==

Please join in the [http://groups.google.com/group/verifybamid VerifyBamID Google Group] to ask / discuss / comment about verifyBamID.

== What's new ==

(2014/02/13)
* Put verifyBamID in github.
* Added PhoneHome/Version Checking to VerifyBamID

(2012/06/20)
* Fixed a bug of incorrect estimate of contamination when --chip-full option was used (Thanks to Richard Smith)
* Fixed a bug of incorrect per-readgroup output in --chip-* parameter

(2012/05/24)
* Fixed a bug of incorrect per-readgroup output (Thanks to Matthew Flickinger)
* '''(IMPORTANT)''' Add an option to remove either side of overlapping fragment. This option is turned on by default, and can be turned off usig --ignoreOverlapPair. If your sequence data has very short insert size, this update may increase the sensitivity of estimated contamination.
* Changes in the directory structure and Makefile

(2012/05/18) The new release of verifyBamID have undergone major change since the last version (as of 2011 April). Here are the highlights
* The genotype / allele frequency file is now based on VCF format rather than PLINK format.
* The reference sequence information is no longer required
* Uses Brent's method for precise estimation of contamination parameters
* Generate the depth distribution statistics.
* Estimated reference-bias parameters (useful mostly for ABI SOLiD sequence data)

== Build verifyBamID ==

The binary download of verifyBamID is available. You may use that version in Ubuntu 64-bit platform.

If you download the source that includes libStatGen:
tar xvf verifyBamIDLibStatGen.#.#.#.tgz
cd verifyBamID_#.#.#
make
Executable: verifyBamID/bin/verifyBamID

If you download the source without libStatGen:
tar xvf verifyBamID-#.#.#.tar.gz
cd verifyBamID-1.1.0
make cloneLib (if ../libStatGen does not exist)
make
Executable: ./bin/verifyBamID

Note that '''make cloneLib''' command will create a directory ../libStatGen under your verifyBamID directory, and '''make''' will create binary of verifyBamID under verifyBamID/bin/

If you have a different version of libStatGen at that path, then skip the cloneLib step. If the libStatGen you want to use is at a different location then update verifyBamID's Makefile.inc. Replace: LIB_PATH_VERIFY_BAM_ID ?= $(LIB_PATH_GENERAL) with
LIB_PATH_VERIFY_BAM_ID = /path/to/libStatGen

verifyBamID is designed to be reasonably portable.

However, since development occurs only on Ubuntu (9.10-13.10) x86 and x64 platforms, and later, there are likely other portability issues.

== Basic Usage ==

A key step in any genetic analysis is to verify whether data being generated matches expectations. ''verifyBamID'' checks whether reads in a BAM file match previous genotypes for a specific sample. In addition, it detects possible sample mixture from population allele frequency only, which can be particularly useful when the genotype data is not available.

Using a mathematical model that relates observed sequence reads to an hypothetical true genotype, ''verifyBamID'' tries to decide whether sequence reads match a particular individual or are more likely to be contaminated (including a small proportion of foreign DNA), derived from a closely related individual, or derived from a completely different individual.

== Basic Usage Example ==

Here is a typical command line:

verifyBamID --vcf [input.vcf] --bam [input.bam] --out [output.prefix] --verbose --ignoreRG

where
[input.bam] is a BAM (Binary Alignment Map) file of a sequence reads
[input.vcf] is input VCF file containing individual genotypes or AF or AC/AN fields in the INFO field. gzipped VCF is also allowed.
[outPrefix] is output prefix of output files - [outPrefix].{selfRG,selfSM,bestRG,bestSM,depthRG,depthSM} will be created.

More detailed description of command line input is below

== Preparing input files ==

verifyBamID requires two input files - VCF file containing external genotypes or allele frequency information, and the BAM file.

=== VCF input genotype file ===

The input VCF file contains (1) external genotype information and/or (2) allele frequency information as AF entry or AC/AN entries in the INFO field. (See [http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-41 | VCF specification] for further details). If neither information is provided, verifyBamID will not work properly.

If external genotype information is provided, sequence+array method will identify contamination and sample swaps by comparing the concordance between the external genotypes and the sequence reads. Additionally, sequence-only method will provide additional contamination estimates by modeling the sequence reads as mixture of two unknown samples based on the allele frequency information in the VCF file.

Input VCF file needs to meet several additional contraints need to meet in order to properly run verifyBamID.
* The VCF is assumed to be well-formed. For example, verifyBamID does not check whether REF allele actually matches with reference sequence.
* The VCF should only contain SNPs. Current version of verifyBamID does not accept INDELs, MNPs, Structural Variations, or other complex variants.
* The individual IDs in the VCF file, must be identical with the individual identifier in the BAM file. Otherwise, --smID option can override the sample ID information of the BAM file to the ID that matches to the individual IDs in the VCF file.
* IMPORTANT : For targeted sequencing data, it is important to subselect the markers to only include on-target markers in the genotype file. Off-target markers are not likely to have multiple non-duplicated reads at the marker position, and it may create artifacts in the analysis due to overlapping fragments.
* Currently, verifyBamID takes only autosomal chromosomes as input VCF.

An example input VCF file (without external genotype) is provided below. Note that AC and AC entries exists in the INFO field for the allele frequency information.

#CHROM POS ID REF ALT QUAL FILTER INFO
20 61651 SNP20-9651 C A . PASS CR=99.86851;GentrainScore=0.7055;HW=0.077647716;AN=2180;AC=11
20 63231 SNP20-11231 T G . PASS CR=99.93036;GentrainScore=0.7837;HW=0.035481825;AN=2182;AC=275
20 63244 rs6139074 A C . PASS CR=98.893394;GentrainScore=0.8001;HW=7.327299E-7;AN=2162;AC=501
20 63799 rs1418258 C T . PASS CR=99.75217;GentrainScore=0.8170;HW=0.6653377;AN=2182;AC=881

=== Input BAM file ===

verifyBamID requires a sorted, indexed, base quality recalibrated, and duplication-marked BAM file. It also requires to contain "@RG" header lines to annotation different readGroups (sequencing runs and lanes). The SM tag in the "@RG" header should match with one of the genotyped sample. Otherwise, verifyBamID may not be able to test whether the sequenced sample matches with genotyped sample, but will try to detect sample mixture from allele frequency, and will try to detect the best-matching sample among the genotyped sample.

== What the default option does ==

The default option of '''verifyBamID''' is the recommended setting for the most sequencing studies to provide a rapid and informative response. The default option provides the following features:
* --free-mix is turned on for estimating contamination using sequence-only method
* --chip-mix is turned on for estimating contamination or swap using sequence+array method, if the external genotype file is provided in the VCF
* --self is turnd on : The default option does not try to compare the sequence reads to identify the best matching individual (which is possible with --best option). It only compares with the external genotypes from the same individual to the sequenced individual.
* --maxDepth 20 is used without --precise option : The default option is intended for whole genome low coverage sequencing. For the targeted exome sequencing, --maxDepth 1000 and --precise is recommended.
* --ignoreRG is not a default option, but a recommended option, when you want to check the contamination for the entire BAM rather than examining each read group separately. This option will increase the computational efficiency especially in the case whether the sequence reads are multiplexed across many sequencing runs.

== Interpreting output files ==

See also [[Understanding VerifyBamID output]].

=== Output files ===
When verifyBamID runs successfully, the following sets of files may be generated.
* [outPrefix].selfSM - Per-sample statistics describing how well the sample matches to the annotated sample.
* [outPrefix].depthSM - The depth distribution of the sequence reads per sample
* [outPrefix].selfRG - Per-readGroup statistics describing how well each lane matches to the annotated sample. (available only without --ignoreRG option)
* [outPrefix].depthRG - The depth distribution of the sequence reads per readGroup. (available only without --ignoreRG option)
* [outPrefix].bestSM - Per-sample best-match statistics with best-matching sample among the genotyped sample (available only with --best option)
* [outPrefix].bestRG - Per-readgroup best-match statistics with best-matching sample among the genotyped sample (available only with --best and without --ignoreRG option)

=== Column information in the output files ===
The .selfSM/.selfRG/.bestSM/.bestRG files have the following 19 columns per sample, or per readgroup (lane).

# SEQ_SM : Sample ID of the sequenced sample. Obtained from @RG header / SM tag in the BAM file
# RG : ReadGroup ID of sequenced lane. For [outPrefix].selfSM and [outPrefix].bestSM, these values are "ALL"
# CHIP_ID : Sample ID compared to in the genotype file. For [outPrefix].selfRG and [outPrefix].selfSM, these values should be identical to [SEQ_SM] or "NA" if the genotype of sequenced samples are unavailable. For [outPrefix].bestRG and [outPrefix].bestSM, these values should be the ID of best-matching sample among the genotype files compared to.
# # SNPs : # of SNPs passing the criteria from the VCF file
# # READS : Total # of reads loaded from the BAM file
# # AVG_DP : Average sequencing depth at the sites in the VCF file
# FREEMIX : Sequence-only estimate of contamination (0-1 scale)
# FREELK1 : Maximum log-likelihood of the sequence reads given estimated contamination under sequence-only method
# FREELK0 : Log-likelihood of the sequence reads given no contamination under sequence-only method
# FREE_RH : Estimated reference bias parameter Pr(refBase|HET) (when --free-refBias or --free-full is used)
# FREE_RA : Estimated reference bias parameter Pr(refBase|HOMALT) (when --free-refBias or --free-full is used)
# CHIPMIX : Sequence+array estimate of contamination (NA if the external genotype is unavailable) (0-1 scale)
# CHIPLK1 : Maximum log-likelihood of the sequence reads given estimated contamination under sequence+array method (NA if the external genotypes are unavailable)
# CHIPLK0 : Log-likelihood of the sequence reads given no contamination under sequence+array method (NA if the external genotypes are unavailable)
# CHIP_RH : Estimated reference bias parameter Pr(refBase|HET) (when --chip-refBias or --chip-full is used)
# CHIP_RA : Estimated reference bias parameter Pr(refBase|HOMALT) (when --chip-refBias or --chip-full is used)
# DPREF : Depth (Coverage) of HomRef site (based on the genotypes of (SELF_SM/BEST_SM), passing mapQ, baseQual, maxDepth thresholds.
# RDPHET : DPHET/DPREF, Relative depth at Heterozygous site.
# RDPALT : DPHET/DPREF, Relative depth at HomAlt site.

=== A guideline to interpret output files ===

verifyBamID provides a series of information that is informative to determine whether the sample is possibly contaminated or swapped, but there is no single criteria that works for every circumstances. There are a few unmodeled factor in the estimation of [SELF-IBD]/[BEST-IBD] and [%MIX], so please note that the MLE estimation may not always exactly match to the true amount of contamination. Here we provide a guideline to flag potentially contaminated/swapped samples

* Each sample or lane can be checked in this way. When [CHIPMIX] >> 0.02 and/or [FREEMIX] >> 0.02, meaning 2% or more of non-reference bases are observed in reference sites, we recommend to examine the data more carefully for the possibility of contamination.
* We recommend to check each lane for the possibility of sample swaps. When [CHIPMIX] ~ 1 AND [FREEMIX] ~ 0, then it is possible that the sample is swapped with another sample. When [CHIPMIX] ~ 0 in .bestSM file, [CHIP_ID] might be actually the swapped sample. Otherwise, the swapped sample may not exist in the genotype data you have compared.
* When genotype data is not available but allele-frequency-based estimates of [FREEMIX] >= 0.03 and [FREELK1]-[FREELK0] is large, then it is possible that the sample is contaminated with other sample. We recommend to use per-sample data rather than per-lane data for checking this for low coverage data, because the inference will be more confident when there are large number of bases with depth 2 or higher.

== Command Line Options ==

The following parameters are available. Ones with "[]" are in effect:

Available Options
Input Files : --vcf [], --bam [], --subset [],
--smID []
VCF analysis options : --genoError [1.0e-03],
--minAF [0.01],
--minCallRate [0.50]
Individuals to compare with chip data : --site, --self, --best
Chip-free optimization options : --free-none, --free-mix [ON],
--free-refBias, --free-full
With-chip optimization options : --chip-none, --chip-mix [ON],
--chip-refBias, --chip-full
BAM analysis options : --ignoreRG, --ignoreOverlapPair,
--noEOF, --precise, --minMapQ [10],
--maxDepth [20], --minQ [13],
--maxQ [40], --grid [0.05]
Modeling Reference Bias : --refRef [1.00], --refHet [0.50],
--refAlt [0.00]
Output options : --out [], --verbose
PhoneHome : --noPhoneHome,
--phoneHomeThinning [50]

Each option provides the following features:
* --vcf : specify required VCF file
* --bam : specify required BAM file (indexed with .bam.bai or .bai file)
* --subset : list of individual IDs to calculate the allele frequency. All individuals are used if unspecified
* --smID : If the individual ID in the BAM file and VCF file does not match, substitute the BAM file's ID into the specified argument
* --genoError : error rate of the external genotype file
* --minAF : minimum allele frequency of the markers to include
* --minAF : minimum call rate of the markers to include
* --site : If set, use only site information in the VCF and do not compare with the actual genotypes
* --self : Only compare the ID-matching individuals between the VCF and BAM file
* --best : Find the best matching individuals (.bestSM and .bestRG files will be produced). This option is substantially longer than the default option
* --free-none : Do not perform sequence-only method to estimate parameters
* --free-mix : (default) Estimate contamination using sequence-only method with Brent's single dimensional optimization.
* --free-refBias : Estimate the reference bias parameters using sequence-only method with Simplex method
* --free-full : Estimate both reference bias parameters and the contamination parameters using sequence-only method
* --chip-none : Do not perform sequence+array method to estimate parameters
* --free-mix : (default) Estimate contamination using sequence+array method with Brent's single dimensional optimization.
* --free-refBias : Estimate the refernece bias parameters using sequence+array method with Simplex method
* --free-full : Estimate both reference bias parameters and the contamination parameters using sequence+array method
* --ignoreRG : ignore the read grouup level comparison and compare samples only (recommended for an expedited run)
* --ignoreOverlapPair : ignore overlapping pair end fragment covering the same base. Disabling this option may decrease the sensitivity of the method when the insert size is short (with slight gain in the computational speed)
* --noEOF : do not check the EOF marker of the BAM file (for earlier version of BAM)
* --precise : calculate the likelihood in log-scale for high-depth data (recommended when --maxDepth is greater than 20. Can be a little bit slower)
* --minMapQ : minimum mapping quality of the sequence reads to compare
* --minQ : minimum base quality to include
* --maxQ : maximum base quality to cap
* --grid : the grid interval to search the optimum before running Brent's algorithm.
* --refRef : Initial Pr(refBase|HOMREFGeno) parameter
* --refHet : Initial Pr(refBase|HETGeno) parameter
* --refAlt : Initial Pr(refBase|HOMALTGeno) parameter
* --out : output file prefix (required)
* --verbose : print the progress of the method on the screeen
{{PhoneHomeParameters|hdr=====|bullet=1}}

== Principle of Operation ==

Each read group in a BAM file is evaluated independently. This means that in file with multiple read groups, problems will be flagged at the read group level (a plus). However, it also means that it might be hard to discern the correct assignment of read groups with very little data.

For each aligned base that overlaps a known genotype, we calculate the probability the probability that it was derived from a particular known genotype. This comparison considers only bases that overlap previously known genotypes and that meet the base quality and mapping quality thresholds.

Each individual in a pedigree has a different combination of genotypes, and bamGenotypeCheck will systematically search for the individual whose genotypes best match the observed read data.

For more about the technical details, see the page [[Verifying Sample Identities - Implementation]]

== Reference ==

Please cite the following paper:

G. Jun, M. Flickinger, K. N. Hetrick, Kurt, J. M. Romm, K. F. Doheny, G. Abecasis, M. Boehnke,and H. M. Kang, ''Detecting and Estimating Contamination of Human DNA Samples in Sequencing and Array-Based Genotype Data'', American journal of human genetics doi:10.1016/j.ajhg.2012.09.004 (volume 91 issue 5 pp.839 - 848)

== Contamination in Array Data ==

[[VerifyIDintensity]] or [[BAFRegress]] can estimate sample contamination from Illumina genotype array data.

== Acknowledgements ==

VerifyBamID is a result from collaborative effort by Hyun Min Kang, Goo Jun, Matthew Flickinger, Mary Kate Wing, and Goncalo Abecasis. Please email to Hyun Min Kang [[mailto:hmkang@umich.edu| hmkang@umich.edu ]] for any questions.

GotCloud: Alignment Sub-Pipelines

2015-03-19T01:37:18Z

Kleckner: /* Example Command Line */

Back to parent: [[GotCloud]]

== List of Alignment Sub-Pipelines ==

===recab===

This sub-pipeline takes in a list of bam files for each sample, merges the BAMs for samples that have multiple BAMs, dedups and recalibrates, and then indexes the recalibrated BAM.

===recabQC===

This sub-pipeline does everything that *recab* does (takes in a list of bam files for each sample, merges the BAMs for samples that have multiple BAMs, dedups and recalibrates, and then indexes the recalibrated BAM). It then goes the next step to perform quality control (running qplot and verifyBamID).

===bamQC===

This sub-pipeline takes in a single, recalibrated BAM file and its index file (.bai) and performs quality control (running qplot and verifyBamID). It differs from *bamQC_createIndex* in that it requires that the user already have .bai files for the recalibrated BAM files.

===bamQC_createIndex===

This sub-pipeline takes in a single, recalibrated BAM file, creates an index file for it, and performs quality control (running qplot and verifyBamID). It differs from *bamQC* in that it does not require that the user already have a .bai file for the recalibrated BAM file.

== recab ==
*What it does:
# merge BAMs for samples that have multiple BAMs
# dedup and recalibrate
# index the recalibrated BAM

====Inputs====
* Bam files (stored in a [[#BAM_LIST File for recab|BAM_LIST File]])
* Reference files
* (Optional) configuration file to override default options

=====BAM_LIST File for recab=====
* Each line of the BAM list file represents a single individual

Columns:
# sample id
# comma separated population labels (optional column)
# BAM File 1 (preferable to have full paths to BAM files)
# BAM File 2 (if more than 1 BAM per sample)
:...

: # BAM File N (if more than 1 BAM per sample)
[SAMPLE_ID] [COMMA SEPARATED POPULATION LABELS] [BAM_FILE1] [BAM_FILE2] ...
or
[SAMPLE_ID] [BAM_FILE1] [BAM_FILE2] ...

* Notes:
** tab delimited
** multiple BAMs per individual may be provided, but should all be on the same line of the list file
** population label is optional - it will default to <code>ALL</code>
*** only used by Thunder (part of ldrefine pipeline)
*** if all samples are from the same population, population label can be skipped or you can just specify <code>ALL</code> for the population label for each sample.

====Outputs====
Upon successful completion of the *recab* sub-pipeline, you should see the following files/subdirectories under the user specified output directory:
*'''recab/mergedBams/'''
** '''''*/SAMPLE.merged.bam'' - a merged BAM file'''
** ''*/SAMPLE.merged.bam.log'' - merge log
** ''*/SAMPLE.merged.bam.OK'' - temp file indicating the merge step completed successfully

* '''recab/'''
** '''''*/SAMPLE.recal.bam'' - a merged, recalibrated, and deduped BAM file'''
** '''''*/SAMPLE.recal.bam.bai'' - an indexed version of the merged, recalibrated, and deduped BAM file'''
** ''*/SAMPLE.recal.bam.metrics'' - dedup & recalibration log
** ''*/SAMPLE.recal.bam.qemp'' - recalibration tables
** ''*/SAMPLE.recal.bam.done'' - temp file indicating the recalibration step completed successfully
** ''*/SAMPLE.recal.bam.bai.done'' - temp file indicating the indexing step completed successfully
You should see .done and .OK files for each SAMPLE in the index file. If you do not see the .done and .OK files, then your *recab* sub-pipeline failed.

'''On success, the recab/ folder contains the final BAMs and bais.'''

===Command-Line and Configuration Options===

*Required Options
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --list/--bam_list/--bamlist ''file'' || BAM_LIST || path to the [[#BAM_LIST File for recab|BAM_LIST File]] || $(OUT_DIR)/bam.list
|-
| --numjobs ''#'' || || number of jobs to run in parallel || 0 (generate Makefile of steps, but do not run)
|}

*Common Options

{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --outdir ''path'' || OUT_DIR || output directory ||
|-
| --conf ''file'' || || configuration file to use ||
|-
| || REF_DIR || where the reference/resource files are stored || gotcloud.ref subdirectory within the base GotCloud directory
|-
| || REF || [[GotCloud: Genetic Reference and Resource Files#Reference fasta Files|Reference fasta Files]] || $(REF_DIR)/human.g1k.v37.fa
|-
| || DBSNP_VCF || [[GotCloud: Genetic Reference and Resource Files#DBSNP VCF Files|DBSNP VCF Files]] || $(REF_DIR)/dbsnp_135.b37.vcf.gz
|}

==== Example Configuration File ====
Example configuration file where reference files happen to be stored in /path/reference, and bam list file is stored in in path/freeze5
BAM_LIST = /path/freeze5.bam.list
OUT_DIR = /path/freeze5/output
REF_DIR = /path/reference/
REF = $(REF_DIR)/hs37d5.fa
DBSNP_VCF = $(REF_DIR)/dbsnp_135.b37.sites.vcf.gz

==== Example Command Line ====
gotcloud pipe –-name recab --numjobs <N>

== recabQC ==
*What it does:
# merge BAMs for samples that have multiple BAMs
# dedup and recalibrate
# index the recalibrated BAM
# qplot
# verifyBamID

====Inputs====
* Bam files (stored in a [[#BAM_LIST File for recabQC|BAM_LIST]] file)
* Reference files
* (Optional) configuration file to override default options

=====BAM_LIST File for recabQC=====
* Each line of the BAM list file represents a single individual

Columns:
# sample id
# comma separated population labels (optional column)
# BAM File 1 (preferable to have full paths to BAM files)
# BAM File 2 (if more than 1 BAM per sample)
:...

: # BAM File N (if more than 1 BAM per sample)
[SAMPLE_ID] [COMMA SEPARATED POPULATION LABELS] [BAM_FILE1] [BAM_FILE2] ...
or
[SAMPLE_ID] [BAM_FILE1] [BAM_FILE2] ...

* Notes:
** tab delimited
** multiple BAMs per individual may be provided, but should all be on the same line of the list file
** population label is optional - it will default to <code>ALL</code>
*** only used by Thunder (part of ldrefine pipeline)
*** if all samples are from the same population, population label can be skipped or you can just specify <code>ALL</code> for the population label for each sample.

====Outputs====
Upon successful completion of the *recabQC* sub-pipeline, you should see the following files/subdirectories under the user specified output directory:
*'''recab/mergedBams/''' - contains merge results
** '''''*/SAMPLE.merged.bam'' - a merged BAM file'''
** ''*/SAMPLE.merged.bam.log'' - merge log
** ''*/SAMPLE.merged.bam.OK'' - temp file indicating the merge step completed successfully

* '''recab/''' - contains recalibration results
** '''''*/SAMPLE.recal.bam'' - a merged, recalibrated, and deduped BAM file'''
** '''''*/SAMPLE.recal.bam.bai'' - an indexed version of the merged, recalibrated, and deduped BAM file'''
** ''*/SAMPLE.recal.bam.metrics'' - dedup & recalibration log
** ''*/SAMPLE.recal.bam.qemp'' - recalibration tables
** ''*/SAMPLE.recal.bam.done'' - temp file indicating the recalibration step completed successfully
** ''*/SAMPLE.recal.bam.bai.done'' - temp file indicating the indexing step completed successfully

* '''QCFiles/''' - contains quality control results
** VerifyBamID Output - see [[VerifyBamID#A_guideline_to_interpret_output_files|VerifyBamID: A guideline to interpret output files]] for more information
*** ''*/SAMPLE.genoCheck.depthRG'' - depth distribution of the sequence reads per read group
*** ''*/SAMPLE.genoCheck.depthSM'' - depth distribution of the sequence reads per sample
*** ''*/SAMPLE.genoCheck.err'' - log file
*** ''*/SAMPLE.genoCheck.log'' - log file
*** ''*/SAMPLE.genoCheck.OK'' - temp file indicating the VerifyBAMID step completed successfully
*** ''*/SAMPLE.genoCheck.selfRG'' - per-readGroup statistics describing how well each lane matches to the annotated sample
*** '''''*/SAMPLE.genoCheck.selfSM'' - main output file containing the contamination estimate; per-sample statistics describing how well the sample matches to the annotated sample'''
**** Check the 'FREEMIX' column for genotype-free estimate of contamination 0-1 scale, the lower, the better
**** If [FREEMIX] >= 0.03 and [FREELK1]-[FREELK0] is large, possible contamination
** Qplot Output - see: [[QPLOT#Diagnose_sequencing_quality|QPLOT: Diagnose sequencing quality]] for more info on how to use QPLOT results
*** ''*/SAMPLE.qplot.OK'' - temp file indicating the qplot step completed successfully
*** '''''*/SAMPLE.qplot.R'' - Rscript that can be used to generate the pdf graphs'''
*** '''''*/SAMPLE.qplot.stats'' - sample statistics'''

You should see .done and .OK files for each SAMPLE in the index file. If you do not see the .done and .OK files, then your *recabQC* sub-pipeline failed.

'''On success, the recab/ folder contains the final BAMs and bais, while the QCFiles/ folder contains the quality control output'''

===Command-Line and Configuration Options===

*Required Options
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --list/--bam_list/--bamlist ''file'' || BAM_LIST || path to the [[#BAM_LIST File for recabQC|BAM_LIST File]] || $(OUT_DIR)/bam.list
|-
| --numjobs ''#'' || || number of jobs to run in parallel || 0 (generate Makefile of steps, but do not run)
|}

*Common Options

{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --outdir ''path'' || OUT_DIR || output directory ||
|-
| --conf ''file'' || || configuration file to use ||
|-
| || REF_DIR || where the reference/resource files are stored || gotcloud.ref subdirectory within the base GotCloud directory
|-
| || REF || [[GotCloud: Genetic Reference and Resource Files#Reference fasta Files|Reference fasta Files]] || $(REF_DIR)/human.g1k.v37.fa
|-
| || DBSNP_VCF || [[GotCloud: Genetic Reference and Resource Files#DBSNP VCF File|DBSNP VCF Files]] || $(REF_DIR)/dbsnp_135.b37.vcf.gz
|-
| || HM3_VCF || [[GotCloud: Genetic Reference and Resource Files#HapMap3 VCF File|HapMap3 VCF Files]] || $(REF_DIR)/hapmap_3.3.b37.sites.vcf.gz
|}

==== Example Configuration File ====
Example configuration file where reference files happen to be stored in /path/reference, and bam list file is stored in in path/freeze5
BAM_LIST = /path/freeze5.bam.list
OUT_DIR = /path/freeze5/output
REF_DIR = /path/reference/
REF = $(REF_DIR)/hs37d5.fa
DBSNP_VCF = $(REF_DIR)/dbsnp_135.b37.sites.vcf.gz
HM3_VCF = $(REF_DIR)/hapmap3_r3_b37.sites.vcf.gz

==== Example Command Line ====
gotcloud pipe –-name recabQC --numjobs <N>

== bamQC ==
*What it does:
# qplot
# verifyBamID

====Inputs====
* Single merged, recalibrated, and deduped BAM file for each subject (stored in a [[#BAM_LIST File for bamQC|BAM_LIST File]])
* BAI file for each subject
* Reference files
* (Optional) configuration file to override default options

=====BAM_LIST File for bamQC=====
* Each line of the BAM list file represents a single individual

Columns:
# sample id
# comma separated population labels (optional column)
# BAM File (preferable to have full path to BAM file)
# BAI File (preferable to have full path to BAI file)

[SAMPLE_ID] [COMMA SEPARATED POPULATION LABELS] [BAM_FILE] [BAI_FILE]
or
[SAMPLE_ID] [BAM_FILE] [BAI_FILE]

* Notes:
** tab delimited
** population label is optional - it will default to <code>ALL</code>
*** only used by Thunder (part of ldrefine pipeline)
*** if all samples are from the same population, population label can be skipped or you can just specify <code>ALL</code> for the population label for each sample.

====Outputs====
Upon successful completion of the *bamQC* sub-pipeline, you should see the following files/subdirectories under the user specified output directory:

* '''QCFiles/''' - contains quality control results
** VerifyBamID Output - see [[VerifyBamID#A_guideline_to_interpret_output_files|VerifyBamID: A guideline to interpret output files]] for more information
*** ''*/SAMPLE.genoCheck.depthRG'' - depth distribution of the sequence reads per read group
*** ''*/SAMPLE.genoCheck.depthSM'' - depth distribution of the sequence reads per sample
*** ''*/SAMPLE.genoCheck.err'' - log file
*** ''*/SAMPLE.genoCheck.log'' - log file
*** ''*/SAMPLE.genoCheck.OK'' - temp file indicating the VerifyBAMID step completed successfully
*** ''*/SAMPLE.genoCheck.selfRG'' - per-readGroup statistics describing how well each lane matches to the annotated sample
*** '''''*/SAMPLE.genoCheck.selfSM'' - main output file containing the contamination estimate; per-sample statistics describing how well the sample matches to the annotated sample'''
**** Check the 'FREEMIX' column for genotype-free estimate of contamination 0-1 scale, the lower, the better
**** If [FREEMIX] >= 0.03 and [FREELK1]-[FREELK0] is large, possible contamination
** Qplot Output - see: [[QPLOT#Diagnose_sequencing_quality|QPLOT: Diagnose sequencing quality]] for more info on how to use QPLOT results
*** ''*/SAMPLE.qplot.OK'' - temp file indicating the qplot step completed successfully
*** '''''*/SAMPLE.qplot.R'' - Rscript that can be used to generate the pdf graphs'''
*** '''''*/SAMPLE.qplot.stats'' - sample statistics'''

You should see .done and .OK files for each SAMPLE in the index file. If you do not see the .done and .OK files, then your *bamQC* sub-pipeline failed.

'''On success, the QCFiles/ folder contains the quality control output'''

===Command-Line and Configuration Options===

*Required Options
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --list/--bam_list/--bamlist ''file'' || BAM_LIST || path to the [[#BAM_LIST File for bamQC|BAM_LIST File]] || $(OUT_DIR)/bam.list
|-
| --numjobs ''#'' || || number of jobs to run in parallel || 0 (generate Makefile of steps, but do not run)
|}

*Common Options

{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --outdir ''path'' || OUT_DIR || output directory ||
|-
| --conf ''file'' || || configuration file to use ||
|-
| || REF_DIR || where the reference/resource files are stored || gotcloud.ref subdirectory within the base GotCloud directory
|-
| || REF || [[GotCloud: Genetic Reference and Resource Files#Reference fasta Files|Reference fasta Files]] || $(REF_DIR)/human.g1k.v37.fa
|-
| || DBSNP_VCF || [[GotCloud: Genetic Reference and Resource Files#DBSNP VCF File|DBSNP VCF Files]] || $(REF_DIR)/dbsnp_135.b37.vcf.gz
|-
| || HM3_VCF || [[GotCloud: Genetic Reference and Resource Files#HapMap3 VCF File|HapMap3 VCF Files]] || $(REF_DIR)/hapmap_3.3.b37.sites.vcf.gz
|}

==== Example Configuration File ====
Example configuration file where reference files happen to be stored in /path/reference, and bam list file is stored in in path/freeze5
BAM_LIST = /path/freeze5.bam.list
OUT_DIR = /path/freeze5/output
REF_DIR = /path/reference/
REF = $(REF_DIR)/hs37d5.fa
DBSNP_VCF = $(REF_DIR)/dbsnp_135.b37.sites.vcf.gz
HM3_VCF = $(REF_DIR)/hapmap3_r3_b37.sites.vcf.gz

==== Example Command Line ====
gotcloud pipe –-name bamQC --numjobs <N>

== bamQC_createIndex ==
*What it does:
# creates a BAI file for any BAM that is missing it
# qplot
# verifyBamID

====Inputs====
* Single merged, recalibrated, and deduped BAM file for each subject (stored in a [[#BAM_LIST File for bamQC_createIndex|BAM_LIST File]])
* Reference files
* (Optional) configuration file to override default options

=====BAM_LIST File for bamQC_createIndex=====
* Each line of the BAM list file represents a single individual

Columns:
# sample id
# comma separated population labels (optional column)
# BAM File (preferable to have full paths to BAM files)

[SAMPLE_ID] [COMMA SEPARATED POPULATION LABELS] [BAM_FILE]
or
[SAMPLE_ID] [BAM_FILE]

* Notes:
** tab delimited
** population label is optional - it will default to <code>ALL</code>
*** only used by Thunder (part of ldrefine pipeline)
*** if all samples are from the same population, population label can be skipped or you can just specify <code>ALL</code> for the population label for each sample.

====Outputs====
Upon successful completion of the *bamQC_createIndex* sub-pipeline, you should see the following files/subdirectories under the user specified output directory:
* A BAI file with the exact same path and name as the BAM file that was input, with *.bai on the end
* '''QCFiles/''' - contains quality control results
** VerifyBamID Output - see [[VerifyBamID#A_guideline_to_interpret_output_files|VerifyBamID: A guideline to interpret output files]] for more information
*** ''*/SAMPLE.genoCheck.depthRG'' - depth distribution of the sequence reads per read group
*** ''*/SAMPLE.genoCheck.depthSM'' - depth distribution of the sequence reads per sample
*** ''*/SAMPLE.genoCheck.err'' - log file
*** ''*/SAMPLE.genoCheck.log'' - log file
*** ''*/SAMPLE.genoCheck.OK'' - temp file indicating the VerifyBAMID step completed successfully
*** ''*/SAMPLE.genoCheck.selfRG'' - per-readGroup statistics describing how well each lane matches to the annotated sample
*** '''''*/SAMPLE.genoCheck.selfSM'' - main output file containing the contamination estimate; per-sample statistics describing how well the sample matches to the annotated sample'''
**** Check the 'FREEMIX' column for genotype-free estimate of contamination 0-1 scale, the lower, the better
**** If [FREEMIX] >= 0.03 and [FREELK1]-[FREELK0] is large, possible contamination
** Qplot Output - see: [[QPLOT#Diagnose_sequencing_quality|QPLOT: Diagnose sequencing quality]] for more info on how to use QPLOT results
*** ''*/SAMPLE.qplot.OK'' - temp file indicating the qplot step completed successfully
*** '''''*/SAMPLE.qplot.R'' - Rscript that can be used to generate the pdf graphs'''
*** '''''*/SAMPLE.qplot.stats'' - sample statistics'''

You should see .done and .OK files for each SAMPLE in the index file. If you do not see the .done and .OK files, then your *bamQC_createIndex* sub-pipeline failed.

'''On success, the QCFiles/ folder contains the quality control output'''

===Command-Line and Configuration Options===

*Required Options
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --list/--bam_list/--bamlist ''file'' || BAM_LIST || path to the [[#BAM_LIST File for bamQC_createIndex|BAM_LIST File]] || $(OUT_DIR)/bam.list
|-
| --numjobs ''#'' || || number of jobs to run in parallel || 0 (generate Makefile of steps, but do not run)
|}

*Common Options

{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --outdir ''path'' || OUT_DIR || output directory ||
|-
| --conf ''file'' || || configuration file to use ||
|-
| || REF_DIR || where the reference/resource files are stored || gotcloud.ref subdirectory within the base GotCloud directory
|-
| || REF || [[GotCloud: Genetic Reference and Resource Files#Reference fasta Files|Reference fasta Files]] || $(REF_DIR)/human.g1k.v37.fa
|-
| || DBSNP_VCF || [[GotCloud: Genetic Reference and Resource Files#DBSNP VCF File|DBSNP VCF Files]] || $(REF_DIR)/dbsnp_135.b37.vcf.gz
|-
| || HM3_VCF || [[GotCloud: Genetic Reference and Resource Files#HapMap3 VCF File|HapMap3 VCF Files]] || $(REF_DIR)/hapmap_3.3.b37.sites.vcf.gz
|}

==== Example Configuration File ====
Example configuration file where reference files happen to be stored in /path/reference, and bam list file is stored in in path/freeze5
BAM_LIST = /path/freeze5.bam.list
OUT_DIR = /path/freeze5/output
REF_DIR = /path/reference/
REF = $(REF_DIR)/hs37d5.fa
DBSNP_VCF = $(REF_DIR)/dbsnp_135.b37.sites.vcf.gz
HM3_VCF = $(REF_DIR)/hapmap3_r3_b37.sites.vcf.gz

==== Example Command Line ====
gotcloud pipe –-name bamQC_createIndex --numjobs <N>

GotCloud: Alignment Sub-Pipelines

2015-03-19T01:36:09Z

Kleckner: /* Command-Line and Configuration Options */

GotCloud: Alignment Sub-Pipelines

2015-03-19T01:35:58Z

Kleckner: /* Inputs */

Back to parent: [[GotCloud]]

== List of Alignment Sub-Pipelines ==

===recab===

This sub-pipeline takes in a list of bam files for each sample, merges the BAMs for samples that have multiple BAMs, dedups and recalibrates, and then indexes the recalibrated BAM.

===recabQC===

This sub-pipeline does everything that *recab* does (takes in a list of bam files for each sample, merges the BAMs for samples that have multiple BAMs, dedups and recalibrates, and then indexes the recalibrated BAM). It then goes the next step to perform quality control (running qplot and verifyBamID).

===bamQC===

This sub-pipeline takes in a single, recalibrated BAM file and its index file (.bai) and performs quality control (running qplot and verifyBamID). It differs from *bamQC_createIndex* in that it requires that the user already have .bai files for the recalibrated BAM files.

===bamQC_createIndex===

This sub-pipeline takes in a single, recalibrated BAM file, creates an index file for it, and performs quality control (running qplot and verifyBamID). It differs from *bamQC* in that it does not require that the user already have a .bai file for the recalibrated BAM file.

== recab ==
*What it does:
# merge BAMs for samples that have multiple BAMs
# dedup and recalibrate
# index the recalibrated BAM

====Inputs====
* Bam files (stored in a [[#BAM_LIST File for recab|BAM_LIST File]])
* Reference files
* (Optional) configuration file to override default options

=====BAM_LIST File for recab=====
* Each line of the BAM list file represents a single individual

Columns:
# sample id
# comma separated population labels (optional column)
# BAM File 1 (preferable to have full paths to BAM files)
# BAM File 2 (if more than 1 BAM per sample)
:...

: # BAM File N (if more than 1 BAM per sample)
[SAMPLE_ID] [COMMA SEPARATED POPULATION LABELS] [BAM_FILE1] [BAM_FILE2] ...
or
[SAMPLE_ID] [BAM_FILE1] [BAM_FILE2] ...

* Notes:
** tab delimited
** multiple BAMs per individual may be provided, but should all be on the same line of the list file
** population label is optional - it will default to <code>ALL</code>
*** only used by Thunder (part of ldrefine pipeline)
*** if all samples are from the same population, population label can be skipped or you can just specify <code>ALL</code> for the population label for each sample.

====Outputs====
Upon successful completion of the *recab* sub-pipeline, you should see the following files/subdirectories under the user specified output directory:
*'''recab/mergedBams/'''
** '''''*/SAMPLE.merged.bam'' - a merged BAM file'''
** ''*/SAMPLE.merged.bam.log'' - merge log
** ''*/SAMPLE.merged.bam.OK'' - temp file indicating the merge step completed successfully

* '''recab/'''
** '''''*/SAMPLE.recal.bam'' - a merged, recalibrated, and deduped BAM file'''
** '''''*/SAMPLE.recal.bam.bai'' - an indexed version of the merged, recalibrated, and deduped BAM file'''
** ''*/SAMPLE.recal.bam.metrics'' - dedup & recalibration log
** ''*/SAMPLE.recal.bam.qemp'' - recalibration tables
** ''*/SAMPLE.recal.bam.done'' - temp file indicating the recalibration step completed successfully
** ''*/SAMPLE.recal.bam.bai.done'' - temp file indicating the indexing step completed successfully
You should see .done and .OK files for each SAMPLE in the index file. If you do not see the .done and .OK files, then your *recab* sub-pipeline failed.

'''On success, the recab/ folder contains the final BAMs and bais.'''

===Command-Line and Configuration Options===

*Required Options
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --list/--bam_list/--bamlist ''file'' || BAM_LIST || path to the [[#BAM_LIST File for recab|BAM_LIST File]] || $(OUT_DIR)/bam.list
|-
| --numjobs ''#'' || || number of jobs to run in parallel || 0 (generate Makefile of steps, but do not run)
|}

*Common Options

{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --outdir ''path'' || OUT_DIR || output directory ||
|-
| --conf ''file'' || || configuration file to use ||
|-
| || REF_DIR || where the reference/resource files are stored || gotcloud.ref subdirectory within the base GotCloud directory
|-
| || REF || [[GotCloud: Genetic Reference and Resource Files#Reference fasta Files|Reference fasta Files]] || $(REF_DIR)/human.g1k.v37.fa
|-
| || DBSNP_VCF || [[GotCloud: Genetic Reference and Resource Files#DBSNP VCF Files|DBSNP VCF Files]] || $(REF_DIR)/dbsnp_135.b37.vcf.gz
|}

==== Example Configuration File ====
Example configuration file where reference files happen to be stored in /path/reference, and bam list file is stored in in path/freeze5
BAM_LIST = /path/freeze5.bam.list
OUT_DIR = /path/freeze5/output
REF_DIR = /path/reference/
REF = $(REF_DIR)/hs37d5.fa
DBSNP_VCF = $(REF_DIR)/dbsnp_135.b37.sites.vcf.gz

==== Example Command Line ====
gotcloud pipe –-name recab --numjobs <N>

== recabQC ==
*What it does:
# merge BAMs for samples that have multiple BAMs
# dedup and recalibrate
# index the recalibrated BAM
# qplot
# verifyBamID

====Inputs====
* Bam files (stored in a [[#BAM_LIST File for recabQC|BAM_LIST]] file)
* Reference files
* (Optional) configuration file to override default options

=====BAM_LIST File for recabQC=====
* Each line of the BAM list file represents a single individual

Columns:
# sample id
# comma separated population labels (optional column)
# BAM File 1 (preferable to have full paths to BAM files)
# BAM File 2 (if more than 1 BAM per sample)
:...

: # BAM File N (if more than 1 BAM per sample)
[SAMPLE_ID] [COMMA SEPARATED POPULATION LABELS] [BAM_FILE1] [BAM_FILE2] ...
or
[SAMPLE_ID] [BAM_FILE1] [BAM_FILE2] ...

* Notes:
** tab delimited
** multiple BAMs per individual may be provided, but should all be on the same line of the list file
** population label is optional - it will default to <code>ALL</code>
*** only used by Thunder (part of ldrefine pipeline)
*** if all samples are from the same population, population label can be skipped or you can just specify <code>ALL</code> for the population label for each sample.

====Outputs====
Upon successful completion of the *recabQC* sub-pipeline, you should see the following files/subdirectories under the user specified output directory:
*'''recab/mergedBams/''' - contains merge results
** '''''*/SAMPLE.merged.bam'' - a merged BAM file'''
** ''*/SAMPLE.merged.bam.log'' - merge log
** ''*/SAMPLE.merged.bam.OK'' - temp file indicating the merge step completed successfully

* '''recab/''' - contains recalibration results
** '''''*/SAMPLE.recal.bam'' - a merged, recalibrated, and deduped BAM file'''
** '''''*/SAMPLE.recal.bam.bai'' - an indexed version of the merged, recalibrated, and deduped BAM file'''
** ''*/SAMPLE.recal.bam.metrics'' - dedup & recalibration log
** ''*/SAMPLE.recal.bam.qemp'' - recalibration tables
** ''*/SAMPLE.recal.bam.done'' - temp file indicating the recalibration step completed successfully
** ''*/SAMPLE.recal.bam.bai.done'' - temp file indicating the indexing step completed successfully

* '''QCFiles/''' - contains quality control results
** VerifyBamID Output - see [[VerifyBamID#A_guideline_to_interpret_output_files|VerifyBamID: A guideline to interpret output files]] for more information
*** ''*/SAMPLE.genoCheck.depthRG'' - depth distribution of the sequence reads per read group
*** ''*/SAMPLE.genoCheck.depthSM'' - depth distribution of the sequence reads per sample
*** ''*/SAMPLE.genoCheck.err'' - log file
*** ''*/SAMPLE.genoCheck.log'' - log file
*** ''*/SAMPLE.genoCheck.OK'' - temp file indicating the VerifyBAMID step completed successfully
*** ''*/SAMPLE.genoCheck.selfRG'' - per-readGroup statistics describing how well each lane matches to the annotated sample
*** '''''*/SAMPLE.genoCheck.selfSM'' - main output file containing the contamination estimate; per-sample statistics describing how well the sample matches to the annotated sample'''
**** Check the 'FREEMIX' column for genotype-free estimate of contamination 0-1 scale, the lower, the better
**** If [FREEMIX] >= 0.03 and [FREELK1]-[FREELK0] is large, possible contamination
** Qplot Output - see: [[QPLOT#Diagnose_sequencing_quality|QPLOT: Diagnose sequencing quality]] for more info on how to use QPLOT results
*** ''*/SAMPLE.qplot.OK'' - temp file indicating the qplot step completed successfully
*** '''''*/SAMPLE.qplot.R'' - Rscript that can be used to generate the pdf graphs'''
*** '''''*/SAMPLE.qplot.stats'' - sample statistics'''

You should see .done and .OK files for each SAMPLE in the index file. If you do not see the .done and .OK files, then your *recabQC* sub-pipeline failed.

'''On success, the recab/ folder contains the final BAMs and bais, while the QCFiles/ folder contains the quality control output'''

===Command-Line and Configuration Options===

*Required Options
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --list/--bam_list/--bamlist ''file'' || BAM_LIST || path to the [[#BAM_LIST File for recabQC|BAM_LIST File]] || $(OUT_DIR)/bam.list
|-
| --numjobs ''#'' || || number of jobs to run in parallel || 0 (generate Makefile of steps, but do not run)
|}

*Common Options

{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --outdir ''path'' || OUT_DIR || output directory ||
|-
| --conf ''file'' || || configuration file to use ||
|-
| || REF_DIR || where the reference/resource files are stored || gotcloud.ref subdirectory within the base GotCloud directory
|-
| || REF || [[GotCloud: Genetic Reference and Resource Files#Reference fasta Files|Reference fasta Files]] || $(REF_DIR)/human.g1k.v37.fa
|-
| || DBSNP_VCF || [[GotCloud: Genetic Reference and Resource Files#DBSNP VCF File|DBSNP VCF Files]] || $(REF_DIR)/dbsnp_135.b37.vcf.gz
|-
| || HM3_VCF || [[GotCloud: Genetic Reference and Resource Files#HapMap3 VCF File|HapMap3 VCF Files]] || $(REF_DIR)/hapmap_3.3.b37.sites.vcf.gz
|}

==== Example Configuration File ====
Example configuration file where reference files happen to be stored in /path/reference, and bam list file is stored in in path/freeze5
BAM_LIST = /path/freeze5.bam.list
OUT_DIR = /path/freeze5/output
REF_DIR = /path/reference/
REF = $(REF_DIR)/hs37d5.fa
DBSNP_VCF = $(REF_DIR)/dbsnp_135.b37.sites.vcf.gz
HM3_VCF = $(REF_DIR)/hapmap3_r3_b37.sites.vcf.gz

==== Example Command Line ====
gotcloud pipe –-name recabQC --numjobs <N>

== bamQC ==
*What it does:
# qplot
# verifyBamID

====Inputs====
* Single merged, recalibrated, and deduped BAM file for each subject (stored in a [[#BAM_LIST File for bamQC|BAM_LIST File]])
* BAI file for each subject
* Reference files
* (Optional) configuration file to override default options

=====BAM_LIST File for bamQC=====
* Each line of the BAM list file represents a single individual

Columns:
# sample id
# comma separated population labels (optional column)
# BAM File (preferable to have full path to BAM file)
# BAI File (preferable to have full path to BAI file)

[SAMPLE_ID] [COMMA SEPARATED POPULATION LABELS] [BAM_FILE] [BAI_FILE]
or
[SAMPLE_ID] [BAM_FILE] [BAI_FILE]

* Notes:
** tab delimited
** population label is optional - it will default to <code>ALL</code>
*** only used by Thunder (part of ldrefine pipeline)
*** if all samples are from the same population, population label can be skipped or you can just specify <code>ALL</code> for the population label for each sample.

====Outputs====
Upon successful completion of the *bamQC* sub-pipeline, you should see the following files/subdirectories under the user specified output directory:

* '''QCFiles/''' - contains quality control results
** VerifyBamID Output - see [[VerifyBamID#A_guideline_to_interpret_output_files|VerifyBamID: A guideline to interpret output files]] for more information
*** ''*/SAMPLE.genoCheck.depthRG'' - depth distribution of the sequence reads per read group
*** ''*/SAMPLE.genoCheck.depthSM'' - depth distribution of the sequence reads per sample
*** ''*/SAMPLE.genoCheck.err'' - log file
*** ''*/SAMPLE.genoCheck.log'' - log file
*** ''*/SAMPLE.genoCheck.OK'' - temp file indicating the VerifyBAMID step completed successfully
*** ''*/SAMPLE.genoCheck.selfRG'' - per-readGroup statistics describing how well each lane matches to the annotated sample
*** '''''*/SAMPLE.genoCheck.selfSM'' - main output file containing the contamination estimate; per-sample statistics describing how well the sample matches to the annotated sample'''
**** Check the 'FREEMIX' column for genotype-free estimate of contamination 0-1 scale, the lower, the better
**** If [FREEMIX] >= 0.03 and [FREELK1]-[FREELK0] is large, possible contamination
** Qplot Output - see: [[QPLOT#Diagnose_sequencing_quality|QPLOT: Diagnose sequencing quality]] for more info on how to use QPLOT results
*** ''*/SAMPLE.qplot.OK'' - temp file indicating the qplot step completed successfully
*** '''''*/SAMPLE.qplot.R'' - Rscript that can be used to generate the pdf graphs'''
*** '''''*/SAMPLE.qplot.stats'' - sample statistics'''

You should see .done and .OK files for each SAMPLE in the index file. If you do not see the .done and .OK files, then your *bamQC* sub-pipeline failed.

'''On success, the QCFiles/ folder contains the quality control output'''

===Command-Line and Configuration Options===

*Required Options
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --list/--bam_list/--bamlist ''file'' || BAM_LIST || path to the [[#BAM_LIST File for bamQC|BAM_LIST File]] || $(OUT_DIR)/bam.list
|-
| --numjobs ''#'' || || number of jobs to run in parallel || 0 (generate Makefile of steps, but do not run)
|}

*Common Options

{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --outdir ''path'' || OUT_DIR || output directory ||
|-
| --conf ''file'' || || configuration file to use ||
|-
| || REF_DIR || where the reference/resource files are stored || gotcloud.ref subdirectory within the base GotCloud directory
|-
| || REF || [[GotCloud: Genetic Reference and Resource Files#Reference fasta Files|Reference fasta Files]] || $(REF_DIR)/human.g1k.v37.fa
|-
| || DBSNP_VCF || [[GotCloud: Genetic Reference and Resource Files#DBSNP VCF File|DBSNP VCF Files]] || $(REF_DIR)/dbsnp_135.b37.vcf.gz
|-
| || HM3_VCF || [[GotCloud: Genetic Reference and Resource Files#HapMap3 VCF File|HapMap3 VCF Files]] || $(REF_DIR)/hapmap_3.3.b37.sites.vcf.gz
|}

==== Example Configuration File ====
Example configuration file where reference files happen to be stored in /path/reference, and bam list file is stored in in path/freeze5
BAM_LIST = /path/freeze5.bam.list
OUT_DIR = /path/freeze5/output
REF_DIR = /path/reference/
REF = $(REF_DIR)/hs37d5.fa
DBSNP_VCF = $(REF_DIR)/dbsnp_135.b37.sites.vcf.gz
HM3_VCF = $(REF_DIR)/hapmap3_r3_b37.sites.vcf.gz

==== Example Command Line ====
gotcloud pipe –-name bamQC --numjobs <N>

== bamQC_createIndex ==
*What it does:
# creates a BAI file for any BAM that is missing it
# qplot
# verifyBamID

====Inputs====
* Single merged, recalibrated, and deduped BAM file for each subject (stored in a [[#BAM_LIST File for bamQC_createIndex|BAM_LIST File]])
* Reference files
* (Optional) configuration file to override default options

=====BAM_LIST File for bamQC_createIndex=====
* Each line of the BAM list file represents a single individual

Columns:
# sample id
# comma separated population labels (optional column)
# BAM File (preferable to have full paths to BAM files)

[SAMPLE_ID] [COMMA SEPARATED POPULATION LABELS] [BAM_FILE]
or
[SAMPLE_ID] [BAM_FILE]

* Notes:
** tab delimited
** population label is optional - it will default to <code>ALL</code>
*** only used by Thunder (part of ldrefine pipeline)
*** if all samples are from the same population, population label can be skipped or you can just specify <code>ALL</code> for the population label for each sample.

====Outputs====
Upon successful completion of the *bamQC_createIndex* sub-pipeline, you should see the following files/subdirectories under the user specified output directory:
* A BAI file with the exact same path and name as the BAM file that was input, with *.bai on the end
* '''QCFiles/''' - contains quality control results
** VerifyBamID Output - see [[VerifyBamID#A_guideline_to_interpret_output_files|VerifyBamID: A guideline to interpret output files]] for more information
*** ''*/SAMPLE.genoCheck.depthRG'' - depth distribution of the sequence reads per read group
*** ''*/SAMPLE.genoCheck.depthSM'' - depth distribution of the sequence reads per sample
*** ''*/SAMPLE.genoCheck.err'' - log file
*** ''*/SAMPLE.genoCheck.log'' - log file
*** ''*/SAMPLE.genoCheck.OK'' - temp file indicating the VerifyBAMID step completed successfully
*** ''*/SAMPLE.genoCheck.selfRG'' - per-readGroup statistics describing how well each lane matches to the annotated sample
*** '''''*/SAMPLE.genoCheck.selfSM'' - main output file containing the contamination estimate; per-sample statistics describing how well the sample matches to the annotated sample'''
**** Check the 'FREEMIX' column for genotype-free estimate of contamination 0-1 scale, the lower, the better
**** If [FREEMIX] >= 0.03 and [FREELK1]-[FREELK0] is large, possible contamination
** Qplot Output - see: [[QPLOT#Diagnose_sequencing_quality|QPLOT: Diagnose sequencing quality]] for more info on how to use QPLOT results
*** ''*/SAMPLE.qplot.OK'' - temp file indicating the qplot step completed successfully
*** '''''*/SAMPLE.qplot.R'' - Rscript that can be used to generate the pdf graphs'''
*** '''''*/SAMPLE.qplot.stats'' - sample statistics'''

You should see .done and .OK files for each SAMPLE in the index file. If you do not see the .done and .OK files, then your *bamQC_createIndex* sub-pipeline failed.

'''On success, the QCFiles/ folder contains the quality control output'''

===Command-Line and Configuration Options===

*Required Options
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --list/--bam_list/--bamlist ''file'' || BAM_LIST || path to the [[#BAM_LIST File for bamQC|BAM_LIST File]] || $(OUT_DIR)/bam.list
|-
| --numjobs ''#'' || || number of jobs to run in parallel || 0 (generate Makefile of steps, but do not run)
|}

*Common Options

{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --outdir ''path'' || OUT_DIR || output directory ||
|-
| --conf ''file'' || || configuration file to use ||
|-
| || REF_DIR || where the reference/resource files are stored || gotcloud.ref subdirectory within the base GotCloud directory
|-
| || REF || [[GotCloud: Genetic Reference and Resource Files#Reference fasta Files|Reference fasta Files]] || $(REF_DIR)/human.g1k.v37.fa
|-
| || DBSNP_VCF || [[GotCloud: Genetic Reference and Resource Files#DBSNP VCF File|DBSNP VCF Files]] || $(REF_DIR)/dbsnp_135.b37.vcf.gz
|-
| || HM3_VCF || [[GotCloud: Genetic Reference and Resource Files#HapMap3 VCF File|HapMap3 VCF Files]] || $(REF_DIR)/hapmap_3.3.b37.sites.vcf.gz
|}

==== Example Configuration File ====
Example configuration file where reference files happen to be stored in /path/reference, and bam list file is stored in in path/freeze5
BAM_LIST = /path/freeze5.bam.list
OUT_DIR = /path/freeze5/output
REF_DIR = /path/reference/
REF = $(REF_DIR)/hs37d5.fa
DBSNP_VCF = $(REF_DIR)/dbsnp_135.b37.sites.vcf.gz
HM3_VCF = $(REF_DIR)/hapmap3_r3_b37.sites.vcf.gz

==== Example Command Line ====
gotcloud pipe –-name bamQC --numjobs <N>

GotCloud: Alignment Sub-Pipelines

2015-03-19T01:35:44Z

Kleckner: /* BAM_LIST File for bamQC */

Back to parent: [[GotCloud]]

== List of Alignment Sub-Pipelines ==

===recab===

This sub-pipeline takes in a list of bam files for each sample, merges the BAMs for samples that have multiple BAMs, dedups and recalibrates, and then indexes the recalibrated BAM.

===recabQC===

This sub-pipeline does everything that *recab* does (takes in a list of bam files for each sample, merges the BAMs for samples that have multiple BAMs, dedups and recalibrates, and then indexes the recalibrated BAM). It then goes the next step to perform quality control (running qplot and verifyBamID).

===bamQC===

This sub-pipeline takes in a single, recalibrated BAM file and its index file (.bai) and performs quality control (running qplot and verifyBamID). It differs from *bamQC_createIndex* in that it requires that the user already have .bai files for the recalibrated BAM files.

===bamQC_createIndex===

This sub-pipeline takes in a single, recalibrated BAM file, creates an index file for it, and performs quality control (running qplot and verifyBamID). It differs from *bamQC* in that it does not require that the user already have a .bai file for the recalibrated BAM file.

== recab ==
*What it does:
# merge BAMs for samples that have multiple BAMs
# dedup and recalibrate
# index the recalibrated BAM

====Inputs====
* Bam files (stored in a [[#BAM_LIST File for recab|BAM_LIST File]])
* Reference files
* (Optional) configuration file to override default options

=====BAM_LIST File for recab=====
* Each line of the BAM list file represents a single individual

Columns:
# sample id
# comma separated population labels (optional column)
# BAM File 1 (preferable to have full paths to BAM files)
# BAM File 2 (if more than 1 BAM per sample)
:...

: # BAM File N (if more than 1 BAM per sample)
[SAMPLE_ID] [COMMA SEPARATED POPULATION LABELS] [BAM_FILE1] [BAM_FILE2] ...
or
[SAMPLE_ID] [BAM_FILE1] [BAM_FILE2] ...

* Notes:
** tab delimited
** multiple BAMs per individual may be provided, but should all be on the same line of the list file
** population label is optional - it will default to <code>ALL</code>
*** only used by Thunder (part of ldrefine pipeline)
*** if all samples are from the same population, population label can be skipped or you can just specify <code>ALL</code> for the population label for each sample.

====Outputs====
Upon successful completion of the *recab* sub-pipeline, you should see the following files/subdirectories under the user specified output directory:
*'''recab/mergedBams/'''
** '''''*/SAMPLE.merged.bam'' - a merged BAM file'''
** ''*/SAMPLE.merged.bam.log'' - merge log
** ''*/SAMPLE.merged.bam.OK'' - temp file indicating the merge step completed successfully

* '''recab/'''
** '''''*/SAMPLE.recal.bam'' - a merged, recalibrated, and deduped BAM file'''
** '''''*/SAMPLE.recal.bam.bai'' - an indexed version of the merged, recalibrated, and deduped BAM file'''
** ''*/SAMPLE.recal.bam.metrics'' - dedup & recalibration log
** ''*/SAMPLE.recal.bam.qemp'' - recalibration tables
** ''*/SAMPLE.recal.bam.done'' - temp file indicating the recalibration step completed successfully
** ''*/SAMPLE.recal.bam.bai.done'' - temp file indicating the indexing step completed successfully
You should see .done and .OK files for each SAMPLE in the index file. If you do not see the .done and .OK files, then your *recab* sub-pipeline failed.

'''On success, the recab/ folder contains the final BAMs and bais.'''

===Command-Line and Configuration Options===

*Required Options
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --list/--bam_list/--bamlist ''file'' || BAM_LIST || path to the [[#BAM_LIST File for recab|BAM_LIST File]] || $(OUT_DIR)/bam.list
|-
| --numjobs ''#'' || || number of jobs to run in parallel || 0 (generate Makefile of steps, but do not run)
|}

*Common Options

{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --outdir ''path'' || OUT_DIR || output directory ||
|-
| --conf ''file'' || || configuration file to use ||
|-
| || REF_DIR || where the reference/resource files are stored || gotcloud.ref subdirectory within the base GotCloud directory
|-
| || REF || [[GotCloud: Genetic Reference and Resource Files#Reference fasta Files|Reference fasta Files]] || $(REF_DIR)/human.g1k.v37.fa
|-
| || DBSNP_VCF || [[GotCloud: Genetic Reference and Resource Files#DBSNP VCF Files|DBSNP VCF Files]] || $(REF_DIR)/dbsnp_135.b37.vcf.gz
|}

==== Example Configuration File ====
Example configuration file where reference files happen to be stored in /path/reference, and bam list file is stored in in path/freeze5
BAM_LIST = /path/freeze5.bam.list
OUT_DIR = /path/freeze5/output
REF_DIR = /path/reference/
REF = $(REF_DIR)/hs37d5.fa
DBSNP_VCF = $(REF_DIR)/dbsnp_135.b37.sites.vcf.gz

==== Example Command Line ====
gotcloud pipe –-name recab --numjobs <N>

== recabQC ==
*What it does:
# merge BAMs for samples that have multiple BAMs
# dedup and recalibrate
# index the recalibrated BAM
# qplot
# verifyBamID

====Inputs====
* Bam files (stored in a [[#BAM_LIST File for recabQC|BAM_LIST]] file)
* Reference files
* (Optional) configuration file to override default options

=====BAM_LIST File for recabQC=====
* Each line of the BAM list file represents a single individual

Columns:
# sample id
# comma separated population labels (optional column)
# BAM File 1 (preferable to have full paths to BAM files)
# BAM File 2 (if more than 1 BAM per sample)
:...

: # BAM File N (if more than 1 BAM per sample)
[SAMPLE_ID] [COMMA SEPARATED POPULATION LABELS] [BAM_FILE1] [BAM_FILE2] ...
or
[SAMPLE_ID] [BAM_FILE1] [BAM_FILE2] ...

* Notes:
** tab delimited
** multiple BAMs per individual may be provided, but should all be on the same line of the list file
** population label is optional - it will default to <code>ALL</code>
*** only used by Thunder (part of ldrefine pipeline)
*** if all samples are from the same population, population label can be skipped or you can just specify <code>ALL</code> for the population label for each sample.

====Outputs====
Upon successful completion of the *recabQC* sub-pipeline, you should see the following files/subdirectories under the user specified output directory:
*'''recab/mergedBams/''' - contains merge results
** '''''*/SAMPLE.merged.bam'' - a merged BAM file'''
** ''*/SAMPLE.merged.bam.log'' - merge log
** ''*/SAMPLE.merged.bam.OK'' - temp file indicating the merge step completed successfully

* '''recab/''' - contains recalibration results
** '''''*/SAMPLE.recal.bam'' - a merged, recalibrated, and deduped BAM file'''
** '''''*/SAMPLE.recal.bam.bai'' - an indexed version of the merged, recalibrated, and deduped BAM file'''
** ''*/SAMPLE.recal.bam.metrics'' - dedup & recalibration log
** ''*/SAMPLE.recal.bam.qemp'' - recalibration tables
** ''*/SAMPLE.recal.bam.done'' - temp file indicating the recalibration step completed successfully
** ''*/SAMPLE.recal.bam.bai.done'' - temp file indicating the indexing step completed successfully

* '''QCFiles/''' - contains quality control results
** VerifyBamID Output - see [[VerifyBamID#A_guideline_to_interpret_output_files|VerifyBamID: A guideline to interpret output files]] for more information
*** ''*/SAMPLE.genoCheck.depthRG'' - depth distribution of the sequence reads per read group
*** ''*/SAMPLE.genoCheck.depthSM'' - depth distribution of the sequence reads per sample
*** ''*/SAMPLE.genoCheck.err'' - log file
*** ''*/SAMPLE.genoCheck.log'' - log file
*** ''*/SAMPLE.genoCheck.OK'' - temp file indicating the VerifyBAMID step completed successfully
*** ''*/SAMPLE.genoCheck.selfRG'' - per-readGroup statistics describing how well each lane matches to the annotated sample
*** '''''*/SAMPLE.genoCheck.selfSM'' - main output file containing the contamination estimate; per-sample statistics describing how well the sample matches to the annotated sample'''
**** Check the 'FREEMIX' column for genotype-free estimate of contamination 0-1 scale, the lower, the better
**** If [FREEMIX] >= 0.03 and [FREELK1]-[FREELK0] is large, possible contamination
** Qplot Output - see: [[QPLOT#Diagnose_sequencing_quality|QPLOT: Diagnose sequencing quality]] for more info on how to use QPLOT results
*** ''*/SAMPLE.qplot.OK'' - temp file indicating the qplot step completed successfully
*** '''''*/SAMPLE.qplot.R'' - Rscript that can be used to generate the pdf graphs'''
*** '''''*/SAMPLE.qplot.stats'' - sample statistics'''

You should see .done and .OK files for each SAMPLE in the index file. If you do not see the .done and .OK files, then your *recabQC* sub-pipeline failed.

'''On success, the recab/ folder contains the final BAMs and bais, while the QCFiles/ folder contains the quality control output'''

===Command-Line and Configuration Options===

*Required Options
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --list/--bam_list/--bamlist ''file'' || BAM_LIST || path to the [[#BAM_LIST File for recabQC|BAM_LIST File]] || $(OUT_DIR)/bam.list
|-
| --numjobs ''#'' || || number of jobs to run in parallel || 0 (generate Makefile of steps, but do not run)
|}

*Common Options

{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --outdir ''path'' || OUT_DIR || output directory ||
|-
| --conf ''file'' || || configuration file to use ||
|-
| || REF_DIR || where the reference/resource files are stored || gotcloud.ref subdirectory within the base GotCloud directory
|-
| || REF || [[GotCloud: Genetic Reference and Resource Files#Reference fasta Files|Reference fasta Files]] || $(REF_DIR)/human.g1k.v37.fa
|-
| || DBSNP_VCF || [[GotCloud: Genetic Reference and Resource Files#DBSNP VCF File|DBSNP VCF Files]] || $(REF_DIR)/dbsnp_135.b37.vcf.gz
|-
| || HM3_VCF || [[GotCloud: Genetic Reference and Resource Files#HapMap3 VCF File|HapMap3 VCF Files]] || $(REF_DIR)/hapmap_3.3.b37.sites.vcf.gz
|}

==== Example Configuration File ====
Example configuration file where reference files happen to be stored in /path/reference, and bam list file is stored in in path/freeze5
BAM_LIST = /path/freeze5.bam.list
OUT_DIR = /path/freeze5/output
REF_DIR = /path/reference/
REF = $(REF_DIR)/hs37d5.fa
DBSNP_VCF = $(REF_DIR)/dbsnp_135.b37.sites.vcf.gz
HM3_VCF = $(REF_DIR)/hapmap3_r3_b37.sites.vcf.gz

==== Example Command Line ====
gotcloud pipe –-name recabQC --numjobs <N>

== bamQC ==
*What it does:
# qplot
# verifyBamID

====Inputs====
* Single merged, recalibrated, and deduped BAM file for each subject (stored in a [[#BAM_LIST File for bamQC|BAM_LIST File]])
* BAI file for each subject
* Reference files
* (Optional) configuration file to override default options

=====BAM_LIST File for bamQC=====
* Each line of the BAM list file represents a single individual

Columns:
# sample id
# comma separated population labels (optional column)
# BAM File (preferable to have full path to BAM file)
# BAI File (preferable to have full path to BAI file)

[SAMPLE_ID] [COMMA SEPARATED POPULATION LABELS] [BAM_FILE] [BAI_FILE]
or
[SAMPLE_ID] [BAM_FILE] [BAI_FILE]

* Notes:
** tab delimited
** population label is optional - it will default to <code>ALL</code>
*** only used by Thunder (part of ldrefine pipeline)
*** if all samples are from the same population, population label can be skipped or you can just specify <code>ALL</code> for the population label for each sample.

====Outputs====
Upon successful completion of the *bamQC* sub-pipeline, you should see the following files/subdirectories under the user specified output directory:

* '''QCFiles/''' - contains quality control results
** VerifyBamID Output - see [[VerifyBamID#A_guideline_to_interpret_output_files|VerifyBamID: A guideline to interpret output files]] for more information
*** ''*/SAMPLE.genoCheck.depthRG'' - depth distribution of the sequence reads per read group
*** ''*/SAMPLE.genoCheck.depthSM'' - depth distribution of the sequence reads per sample
*** ''*/SAMPLE.genoCheck.err'' - log file
*** ''*/SAMPLE.genoCheck.log'' - log file
*** ''*/SAMPLE.genoCheck.OK'' - temp file indicating the VerifyBAMID step completed successfully
*** ''*/SAMPLE.genoCheck.selfRG'' - per-readGroup statistics describing how well each lane matches to the annotated sample
*** '''''*/SAMPLE.genoCheck.selfSM'' - main output file containing the contamination estimate; per-sample statistics describing how well the sample matches to the annotated sample'''
**** Check the 'FREEMIX' column for genotype-free estimate of contamination 0-1 scale, the lower, the better
**** If [FREEMIX] >= 0.03 and [FREELK1]-[FREELK0] is large, possible contamination
** Qplot Output - see: [[QPLOT#Diagnose_sequencing_quality|QPLOT: Diagnose sequencing quality]] for more info on how to use QPLOT results
*** ''*/SAMPLE.qplot.OK'' - temp file indicating the qplot step completed successfully
*** '''''*/SAMPLE.qplot.R'' - Rscript that can be used to generate the pdf graphs'''
*** '''''*/SAMPLE.qplot.stats'' - sample statistics'''

You should see .done and .OK files for each SAMPLE in the index file. If you do not see the .done and .OK files, then your *bamQC* sub-pipeline failed.

'''On success, the QCFiles/ folder contains the quality control output'''

===Command-Line and Configuration Options===

*Required Options
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --list/--bam_list/--bamlist ''file'' || BAM_LIST || path to the [[#BAM_LIST File for bamQC|BAM_LIST File]] || $(OUT_DIR)/bam.list
|-
| --numjobs ''#'' || || number of jobs to run in parallel || 0 (generate Makefile of steps, but do not run)
|}

*Common Options

{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --outdir ''path'' || OUT_DIR || output directory ||
|-
| --conf ''file'' || || configuration file to use ||
|-
| || REF_DIR || where the reference/resource files are stored || gotcloud.ref subdirectory within the base GotCloud directory
|-
| || REF || [[GotCloud: Genetic Reference and Resource Files#Reference fasta Files|Reference fasta Files]] || $(REF_DIR)/human.g1k.v37.fa
|-
| || DBSNP_VCF || [[GotCloud: Genetic Reference and Resource Files#DBSNP VCF File|DBSNP VCF Files]] || $(REF_DIR)/dbsnp_135.b37.vcf.gz
|-
| || HM3_VCF || [[GotCloud: Genetic Reference and Resource Files#HapMap3 VCF File|HapMap3 VCF Files]] || $(REF_DIR)/hapmap_3.3.b37.sites.vcf.gz
|}

==== Example Configuration File ====
Example configuration file where reference files happen to be stored in /path/reference, and bam list file is stored in in path/freeze5
BAM_LIST = /path/freeze5.bam.list
OUT_DIR = /path/freeze5/output
REF_DIR = /path/reference/
REF = $(REF_DIR)/hs37d5.fa
DBSNP_VCF = $(REF_DIR)/dbsnp_135.b37.sites.vcf.gz
HM3_VCF = $(REF_DIR)/hapmap3_r3_b37.sites.vcf.gz

==== Example Command Line ====
gotcloud pipe –-name bamQC --numjobs <N>

== bamQC_createIndex ==
*What it does:
# creates a BAI file for any BAM that is missing it
# qplot
# verifyBamID

====Inputs====
* Single merged, recalibrated, and deduped BAM file for each subject (stored in a [[#BAM_LIST File for bamQC|BAM_LIST File]])
* Reference files
* (Optional) configuration file to override default options

=====BAM_LIST File for bamQC_createIndex=====
* Each line of the BAM list file represents a single individual

Columns:
# sample id
# comma separated population labels (optional column)
# BAM File (preferable to have full paths to BAM files)

[SAMPLE_ID] [COMMA SEPARATED POPULATION LABELS] [BAM_FILE]
or
[SAMPLE_ID] [BAM_FILE]

* Notes:
** tab delimited
** population label is optional - it will default to <code>ALL</code>
*** only used by Thunder (part of ldrefine pipeline)
*** if all samples are from the same population, population label can be skipped or you can just specify <code>ALL</code> for the population label for each sample.

====Outputs====
Upon successful completion of the *bamQC_createIndex* sub-pipeline, you should see the following files/subdirectories under the user specified output directory:
* A BAI file with the exact same path and name as the BAM file that was input, with *.bai on the end
* '''QCFiles/''' - contains quality control results
** VerifyBamID Output - see [[VerifyBamID#A_guideline_to_interpret_output_files|VerifyBamID: A guideline to interpret output files]] for more information
*** ''*/SAMPLE.genoCheck.depthRG'' - depth distribution of the sequence reads per read group
*** ''*/SAMPLE.genoCheck.depthSM'' - depth distribution of the sequence reads per sample
*** ''*/SAMPLE.genoCheck.err'' - log file
*** ''*/SAMPLE.genoCheck.log'' - log file
*** ''*/SAMPLE.genoCheck.OK'' - temp file indicating the VerifyBAMID step completed successfully
*** ''*/SAMPLE.genoCheck.selfRG'' - per-readGroup statistics describing how well each lane matches to the annotated sample
*** '''''*/SAMPLE.genoCheck.selfSM'' - main output file containing the contamination estimate; per-sample statistics describing how well the sample matches to the annotated sample'''
**** Check the 'FREEMIX' column for genotype-free estimate of contamination 0-1 scale, the lower, the better
**** If [FREEMIX] >= 0.03 and [FREELK1]-[FREELK0] is large, possible contamination
** Qplot Output - see: [[QPLOT#Diagnose_sequencing_quality|QPLOT: Diagnose sequencing quality]] for more info on how to use QPLOT results
*** ''*/SAMPLE.qplot.OK'' - temp file indicating the qplot step completed successfully
*** '''''*/SAMPLE.qplot.R'' - Rscript that can be used to generate the pdf graphs'''
*** '''''*/SAMPLE.qplot.stats'' - sample statistics'''

You should see .done and .OK files for each SAMPLE in the index file. If you do not see the .done and .OK files, then your *bamQC_createIndex* sub-pipeline failed.

'''On success, the QCFiles/ folder contains the quality control output'''

===Command-Line and Configuration Options===

*Required Options
{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --list/--bam_list/--bamlist ''file'' || BAM_LIST || path to the [[#BAM_LIST File for bamQC|BAM_LIST File]] || $(OUT_DIR)/bam.list
|-
| --numjobs ''#'' || || number of jobs to run in parallel || 0 (generate Makefile of steps, but do not run)
|}

*Common Options

{| class="wikitable" style="margin: 1em 1em 1em 0; background-color: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse;" border="1"
! Command-line Flag !! Configuration Key !! Value Description !! Default Value
|-
| --outdir ''path'' || OUT_DIR || output directory ||
|-
| --conf ''file'' || || configuration file to use ||
|-
| || REF_DIR || where the reference/resource files are stored || gotcloud.ref subdirectory within the base GotCloud directory
|-
| || REF || [[GotCloud: Genetic Reference and Resource Files#Reference fasta Files|Reference fasta Files]] || $(REF_DIR)/human.g1k.v37.fa
|-
| || DBSNP_VCF || [[GotCloud: Genetic Reference and Resource Files#DBSNP VCF File|DBSNP VCF Files]] || $(REF_DIR)/dbsnp_135.b37.vcf.gz
|-
| || HM3_VCF || [[GotCloud: Genetic Reference and Resource Files#HapMap3 VCF File|HapMap3 VCF Files]] || $(REF_DIR)/hapmap_3.3.b37.sites.vcf.gz
|}

==== Example Configuration File ====
Example configuration file where reference files happen to be stored in /path/reference, and bam list file is stored in in path/freeze5
BAM_LIST = /path/freeze5.bam.list
OUT_DIR = /path/freeze5/output
REF_DIR = /path/reference/
REF = $(REF_DIR)/hs37d5.fa
DBSNP_VCF = $(REF_DIR)/dbsnp_135.b37.sites.vcf.gz
HM3_VCF = $(REF_DIR)/hapmap3_r3_b37.sites.vcf.gz

==== Example Command Line ====
gotcloud pipe –-name bamQC --numjobs <N>