3,045
edits
Changes
From Genome Analysis Wiki
Jump to navigationJump to searchno edit summary
* make (make on ubuntu)
* make (make on ubuntu)
* libssl (libssl0.9.8 on ubuntu)
* libssl (libssl0.9.8 on ubuntu)
* gcc 4.6 or newer
* gcc 4.4 or newer
=== Step 1c: Install Example Dataset ===
=== Step 1c: Install Example Dataset ===
Run the alignment pipeline (the example aligns 2 samples) :
Run the alignment pipeline (the example aligns 2 samples) :
$GCHOME/gotcloud align --conf $GCDATA/[[Alignment Configuration File|GBR60align.conf]] --outdir $GCOUT
$GCHOME/gotcloud align --conf $GCDATA/[[#Alignment Configuration File|GBR60align.conf]] --outdir [[#Alignment Output Directory|$GCOUT]]
Upon successful completion of the alignment pipeline (about 1-2 minutes), you will see the following message:
Upon successful completion of the alignment pipeline (about 1-2 minutes), you will see the following message:
== Alignment Pipeline ==
== Alignment Pipeline ==
=== List of Input Files needed ===
=== List of Input Files needed for Alignment ===
The command-line inputs to the tutorial alignment pipeline are:
The command-line inputs to the tutorial alignment pipeline are:
# [[#Alignment Configuration File|Configuration File (--conf)]]
# [[#Alignment Configuration File|Configuration File (--conf)]]
# [[#Alignment Output Directory|Output Directory (--outdir)]]
# [[#Alignment Output Directory|Output Directory (--outdir)]]
#* Directory where the output should be placed.
#* Directory where the output should be placed.
Additional information required to run the alignment pipeline:
Additional information required to run the alignment pipeline:
# Index file of FASTQs
# [[#Alignment FASTQ Index File|Index file of FASTQs]]
# Reference files
# [[#Alignment Reference Files|Alignment Reference Files]]
For the tutorial, these values are specified in the configuration file.
For the tutorial, these values are specified in the configuration file.
This configuration file sets:
This configuration file sets:
* INDEX_FILE - file containing the fastqs to be processed as well as the read group information for these fastqs.
* [[#Alignment FASTQ Index File|INDEX_FILE]] - file containing the fastqs to be processed as well as the read group information for these fastqs.
* Reference Information: see [[#Alignment Reference Files|Alignment Reference Files]] for more information
** AS - assembly value to put in the BAM
The index file and chromosome 20 references used in this tutorial are included with the example data under the $GCDATA directory. The tutorial uses chromosome 20 only references in order to speed the processing time.
When running with your own data, you will need to update the:
* Index File to contain the information for your own fastq files
** See [[#Alignment FASTQ Index File|Alignment FASTQ Index File]] for more information on the contents of the index file.
* The reference files to be whole genome references
** If you are just running chromosome 20, you can use the tutorial references
** Whole genome reference files can be downloaded from [[GotCloudReference]].
** See [[#Alignment Reference Files|Alignment Reference Files]] for more information.
Note: It is recommended that you use absolute paths (full path names, like “/home/mktrost/gotcloudReference” rather than just “gotcloudReference”). This example does not use absolute paths in order to be flexible to where the data is installed, but using relative paths requires it to be run from the correct directory.
=== Reference Files ===
Reference files are required for running both the alignment and variant calling pipelines.
The configuration keys for setting these are:
* FA_REF - Genome sequence reference files (needed for both pipelines)
* DBSNP_VCF – DBSNP site VCF file (needed for both pipelines)
* HM3_VCF - HAPMAP site VCF file (needed for both pipelines)
*** human_g1k_v37_chr20.fa.pac
* INDEL_PREFIX - Indel sites file (need for variant calling pipeline)
*** human_g1k_v37_chr20.fa.rbwt
*** human_g1k_v37_chr20.fa.rpac
** HM3_VCF - hapmap vcf
The index file and chromosome 20 references are included with the example data under the $GCDATA directory. The tutorial uses chromosome 20 only references in order to speed the processing time.
The tutorial configuration file is setup to point to the required chromosome 20 reference files which are included with the tutorial example data in $GCDATA/chr20Ref/.
If you are running more than just chromosome 20, you will need whole genome reference files which can be downloaded from [[GotCloudReference]].
The configuration settings for these files are setup in the default configuration so do not need to be specified. You just need to set REF_DIR in your configuration file to the path where you installed your reference files.
To learn more about the reference files that are required, see [[GotCloud: Reference Files]].
=== Alignment Output Directory ===
=== Alignment Output Directory ===
This setting tells the pipeline where to write the output files.
This setting tells the alignment pipeline where to write the output and intermediate files.
The output directory will be created if necessary and will contain the following Directories/files:
The output directory will be created if it doesn't already exist and will contain the following Directories/files:
* bams - directory containing the final bams/bai files
* bams - directory containing the final bams/bai files
** HG00096.OK - indicates that this sample completed alignment processing
** HG00096.OK - indicates that this sample completed alignment processing
** HG00100.OK - indicates that this sample completed alignment processing
** HG00100.OK - indicates that this sample completed alignment processing
* failLogs - directory containing logs from steps that failed
* failLogs - directory containing logs from steps that failed
** this directory is only created if an error is detected
* Makefiles - directory containing the makefiles with commands for processing each sample
* Makefiles - directory containing the makefiles with commands for processing each sample
** biopipe_HG00096.Makefile
** biopipe_HG00096.Makefile – commands for processing sample HG00096
** biopipe_HG00100.Makefile
** biopipe_HG00100.Makefile – commands for processing sample HG00100
** biopipe_HG00096.Makefile.log – log file from running the associated Makefiles
** biopipe_HG00096.Makefile.log – log file from running the associated Makefile
** biopipe_HG00100.Makefile.log – log file from running the associated Makefiles
** biopipe_HG00100.Makefile.log – log file from running the associated Makefile
* QCFiles - directory containing the QC Results
* QCFiles - directory containing the QC Results
**
**
* tmp - directory containing temporary alignment files
* tmp - directory containing temporary alignment files
**
** bwa.sai.t – contains temporary files for the 1st step of bwa that generates sai files from fastq files
*** fastq
**** *.done – indicator files that the step to generate the file completed
**** HG0096
** alignment.bwa – contains temporary files for the 2nd step of bwa that generates BAM files
*** fastq
**** *.done – indicator files that the step to generate the file completed
** alignment.pol - contains temporary files for the polish bam step that cleans up BAM files
** alignment.dedup – contains temporary files for the deduping step
*** *.done – indicator files that the step to generate the file completed
*** *.metrics – metrics files for the deduping step
** alignment.recal – contains temporary files for the deduping step
*** *.log – contains information about the recalibration step
*** *.qemp – contains the recalibration tables used for recalibrating each BAM file
===Alignment FASTQ Index File===
===Index file===
There are four fastq files in {ROOT_DIR}/test/align/fastq/Sample_1 and four fastq files in {ROOT_DIR}/test/align/fastq/Sample_2, both in paired-end format. Normally, we would need to build an index file for these files. Conveniently, an index file (indexFile.txt) already exists for the automatic test samples. It can be found in {ROOT_DIR}/test/align/, and contains the following information in tab-delimited format:
There are four fastq files in {ROOT_DIR}/test/align/fastq/Sample_1 and four fastq files in {ROOT_DIR}/test/align/fastq/Sample_2, both in paired-end format. Normally, we would need to build an index file for these files. Conveniently, an index file (indexFile.txt) already exists for the automatic test samples. It can be found in {ROOT_DIR}/test/align/, and contains the following information in tab-delimited format:
(More information about: [[Mapping_Pipeline#Sequence_Index_File|the index file]].)
(More information about: [[Mapping_Pipeline#Sequence_Index_File|the index file]].)
A configuration file (umake_test.conf) already exists in {ROOT_DIR}/test/umake/. It contains the following information:
A configuration file (umake_test.conf) already exists in {ROOT_DIR}/test/umake/. It contains the following information:
CHRS = 20
BAM_INDEX = GBR60bam.index
############
# References
REF_ROOT = chr20Ref
#
REF = $(REF_ROOT)/human_g1k_v37_chr20.fa
INDEL_PREFIX = $(REF_ROOT)/1kg.pilot_release.merged.indels.sites.hg19
DBSNP_VCF = $(REF_ROOT)/dbsnp135_chr20.vcf.gz
HM3_VCF = $(REF_ROOT)/hapmap_3.3.b37.sites.chr20.vcf.gz
CHRS = 20
CHRS = 20
(More information about: [[Variant_Calling_Pipeline_(UMAKE)#Configuration_File|the configuration file]], [[Variant_Calling_Pipeline_(UMAKE)#Reference_Files|reference files]].)
(More information about: [[Variant_Calling_Pipeline_(UMAKE)#Configuration_File|the configuration file]], [[Variant_Calling_Pipeline_(UMAKE)#Reference_Files|reference files]].)
