3,045
edits
Changes
From Genome Analysis Wiki
Jump to navigationJump to searchGotCloud: Alignment Pipeline (view source)
Revision as of 02:12, 7 November 2012
, 02:12, 7 November 2012no edit summary
The Mapping Pipeline takes FASTQ files and generates recalibrated BAM files from them.
The Mapping Pipeline takes FASTQ files and generates recalibrated BAM files from them.
Here is an overview of the Mapping Pipeline:
[[File:MappingSteps.png]]
== Input Data:==
== Input Data:==
*Raw Sequence (FASTQ) files
*Raw Sequence (FASTQ) files
*Sequence Index file
*Sequence Index file containing fastqs & RG info
*Reference files
*Reference files
*(Optional) Configuration file to override default options
*(Optional) Configuration file to override default options
These are the FASTQ files that need to be mapped to BAM files.
These are the FASTQ files that need to be mapped to BAM files.
These files are specified in the [[#Sequence Index File|Sequence Index File]].
=== Sequence Index File ===
=== Sequence Index File ===
This file specifies the FASTQ files that need to be processed and the Read Group information for them.
This file specifies the FASTQ files that need to be processed and the Read Group information for them.
This file is specified either via the command line parameter <code>--index_file</code> or via the configuration file setting <code>INDEX_FILE</code>.
The command-line setting takes precedence over the configuration file setting.
The Sequence Index is a tab delimited file that starts with a header line. The columns may be in any order.
The Sequence Index is a tab delimited file that starts with a header line. The columns may be in any order.
The RGID, SAMPLE, LIBRARY, CENTER, and PLATFORM are used to populate the Read Group information for this entry. These fields are optional. Either leave the column header out of the file or specify '.' if the column header exists, but the data is N/A. As long as the RGID field is specified non-N/A fields are added to the BAM file.
The RGID, SAMPLE, LIBRARY, CENTER, and PLATFORM are used to populate the Read Group information for this entry. These fields are optional. Either leave the column header out of the file or specify '.' if the column header exists, but the data is N/A. As long as the RGID field is specified non-N/A fields are added to the BAM file.
MERGE_NAME FASTQ1 FASTQ2 RGID SAMPLE LIBRARY CENTER PLATFORM
Sample1 fastq/S1/F1_R1.fastq.gz fastq/S1/F1_R2.fastq.gz RGID1 SampleID1 Lib1 UM ILLUMINA
Sample1 fastq/S1/F2_R1.fastq.gz fastq/S1/F2_R2.fastq.gz RGID1a SampleID1 Lib1 UM ILLUMINA
Sample2 fastq/S2/F1_R1.fastq.gz fastq/S2/F1_R2.fastq.gz RGID2 SampleID2 Lib2 UM ILLUMINA
Sample2 fastq/S2/F2.fastq.gz . RGID2 SampleID2 Lib2 UM ILLUMINA
The <code>--fastq</code>/<code>FASTQ</code> setting can be used to specify a prefix to the FASTQ1/FASTQ2 file paths that should be applied before using the files.
=== Reference Files ===
=== Reference Files ===
** Configuration Name: PLINK
** Configuration Name: PLINK
=== Configuration File ===
Configuration file contains the run-time options including the software binaries and command line arguments. A default configuration file is automatically loaded. Users may specify their own configuration file specifying just the values different than the defaults. The configuration file is not required if there are no values to override.
=== Configuration File ===
Comments begin with a <code>#</code>
Format: KEY = value
Where KEY is the item being set and value is its new value
See [[#Command-Line Options|Command-Line Options]] for values that can be set eitehr via command line or via configuration.
Note: Command-line options take priority over configuration file settings
==== Required Settings ====
See [[#Reference File|Reference File]] for the required reference file settings.
See [[#Sequence Index File|Sequence Index File]] for how to set the index file either via command line options or via configuration.
See [[#Specifying the Output Directory|Specifying the Output Directory]] for how to set the output directory.
==== Turning Off Optional Steps====
Quality Control steps can be disabled.
To Disable QPLOT, set:
RUN_QPLOT = 0
To Disable VerifyBamID, set:
RUN_VERIFY_BAM_ID = 0
==== Optional Configurable Settings ====
You may want to adjust the amount of memory/threads that are used:
There are additional configurable settings, but these are the ones most likely to be adjusted.
* BWA_THREADS = -t N
** Fill in the N with the number of threads you want BWA to run with, default is 1
* BWA_MAX_MEM = 2000000000
** Maximum amount of memory used by samtools sort after running bwa
* JAVA_MEM = -Xmx4g
** Set the maximum size of the java memory allocation pool. Default is 4g, adjust that as necessary.
== Running the Mapping Pipeline ==
== Running the Mapping Pipeline ==
'''cd ~/myseq'''
=== Command-Line Options ===
* help - print usage
</code>
* test OUTPUT_DIR - run the test example placing the output in a user specified OUTPUT_DIR. No other options are required.
* out_dir OUTPUT_DIR - directory for the output
** May also be specified via OUT_DIR in the configuration file
* conf CONFIG_FILE - configuration file
* index_file INDEX_FILE_NAME - name of the index file
** May also be specified via INDEX_FILE in the configuration file
* ref_dir REFERENCE_DIR - value to set config key REF_DIR to, overriding other values, REF_DIR can then be used inside config files.
** May also be specified via REF_DIR in the configuration file
* fastq FASTQ_PATH - prefix path to the fastq files specified in the INDEX_FILE
** May also be specified via FASTQ in the configuration file
* keepTmp - Do not remove the temporary files (removed by default)
** May also be specified via KEEP_TMP in the configuration file
* numjobs N - Replace N with the number of jobs that should be run in parallel
Note: Command-line options take priority over configuration file settings
The mapping pipeline is currently a 2 step process.
'''Step 1: Generate the Makefiles, 1 per sample'''
Run bin/gen_biopipeline.pl with the appropriate command-line parameters.
Example:
bin/gen_biopipeline.pl --conf config.txt --out_dir output
This step generates 1 Makefile per sample in the output/Makefiles/ directory, but does not run them. These Makefiles contain all of the information to run each sample.
Instructions are printed for running the Makefiles.
'''Step 2: Run the Samples'''
Run make -f on each file in the Makefiles directory. Each Makefile is independent and can be run in parallel and across a cloud.
It is recommended that you redirect stdout and stderr to files to save the results.
On failure, the Makefile should report a message like:
make: *** [...] Error 1
Where ... is filled in with other text indicating what step failed.
On success, you should see the following subdirectories under the user specified output directory:
* alignment_recal/
* Makefiles/
* QCFiles/ (if all quality control is not disabled)
* tmp/
You should see a <code>.OK</code> for each Sample in the index file.
If you do not see these <code>.OK</code> files, then your Mapping Pipeline failed.
On success, the alignment.recal/ directory contains the final BAMs, bais, and md5s.
If processing fails part way through, you can pick up where you left off by rerunning the make command.
