3,045
edits
Changes
From Genome Analysis Wiki
Jump to navigationJump to searchno edit summary
GotCloud and this basic tutorial were presented at the [http://ibg.colorado.edu/dokuwiki/doku.php?id=workshop:2013:announcement 2013 IBG Workshop]. It was presented in two sessions. On Wednesday an overview was presented with steps for running the tutorial data: [[Media:IBG2013GotCloud.pdf|IBG2013GotCloud.pdf]]. On Friday more detail on the input files and what goes into generating the input files was presented: [[Media:GotCloudIBGWorkshop2013Friday.pdf|GotCloudIBGWorkshop2013Friday.pdf]].
GotCloud and this basic tutorial were presented at the [http://ibg.colorado.edu/dokuwiki/doku.php?id=workshop:2013:announcement 2013 IBG Workshop]. It was presented in two sessions. On Wednesday an overview was presented with steps for running the tutorial data: [[Media:IBG2013GotCloud.pdf|IBG2013GotCloud.pdf]]. On Friday more detail on the input files and what goes into generating the input files was presented: [[Media:GotCloudIBGWorkshop2013Friday.pdf|GotCloudIBGWorkshop2013Friday.pdf]].
This tutorial is in the process of being updated for gotcloud version 1.06 (April 17. 2013).
== STEP 1 : Setup GotCloud ==
== STEP 1 : Setup GotCloud ==
[[GotCloud]] has been developed and tested on Linux Ubuntu 12.10 and 12.04.2 LTS but has not been tested on other Linux operating systems. It is not available for Windows. If you do not have your own set of machines to run on, GotCloud is also available for Ubuntu running on the Amazon Elastic Compute Cloud, see [[Amazon_Snapshot]] for more information.
[[GotCloud]] has been developed and tested on Linux Ubuntu 12.10 and 12.04.2 LTS but has not been tested on other Linux operating systems. It is not available for Windows. If you do not have your own set of machines to run on, GotCloud is also available for Ubuntu running on the Amazon Elastic Compute Cloud, see [[Amazon_Snapshot]] for more information.
We will use 3 different directories for this tutorial:
# path to the directory where gotcloud is installed, default ~/gotcloud/
# path to the directory where the example data is installed, default ~/gotcloudExample
# path to your output directory, default ~/gotcloudTutorialOut/
If the directories specified above do not reflect the directories you would like to use, replace their occurrances in the instructions below with the appropriate paths.
If the directories specified above do not reflect where gotcloud and/or the example data is installed or where you want your output to go, then replace those directories with the full paths to the appropriate directories.
=== Step 1b: Install GotCloud ===
=== Step 1a: Install GotCloud ===
In order to run this tutorial, you need to make sure you have GotCloud installed on your system.
In order to run this tutorial, you need to make sure you have GotCloud installed on your system.
Otherwise, you can install it in your own directory:
Otherwise, you can install it in your own directory:
# Create & change to the directory where you want gotcloud installed
# Change to the directory where you want gotcloud/ installed
# Download the gotcloud tar from the ftp site.
# Download the gotcloud tar from the ftp site.
# Extract the tar
# Extract the tar
#* Note: as the source builds, many messages will scroll through your terminal. You may even see some warnings. These messages are normal and expected. As long as the build does not end with an error, you have successfully built the source.
#* Note: as the source builds, many messages will scroll through your terminal. You may even see some warnings. These messages are normal and expected. As long as the build does not end with an error, you have successfully built the source.
cd ~
wget ftp://share.sph.umich.edu/gotcloud/gotcloud_latest.tgz # Download
wget ftp://share.sph.umich.edu/gotcloud/gotcloud_latest.tgz # Download
tar xvf gotcloud_latest.tgz --strip 1 # Extract
tar xvf gotcloud_latest.tgz # Extracts into gotcloud/
cd $GCHOME/src; make # Build source
cd ~/gotcloud/src; make # Build source
GotCloud requires the following tools to be installed.
GotCloud requires the following tools to be installed.
You can run $GCHOME/scripts/check_requirements.sh
You can run ~/gotcloud/scripts/check_requirements.sh
...TBD – put in required programs/tools.
...TBD – put in required programs/tools.
* java (java-common default-jre on ubuntu)
* java (java-common default-jre on ubuntu)
* gcc 4.4 or newer
* gcc 4.4 or newer
=== Step 1c: Install Example Dataset ===
=== Step 1b: Install Example Dataset ===
Our dataset consists of 60 individuals from Great Britain (GBR) sequenced by the 1000 Genomes Project. These individuals have been sequenced to an average depth of about 4x.
Our dataset consists of 60 individuals from Great Britain (GBR) sequenced by the 1000 Genomes Project. These individuals have been sequenced to an average depth of about 4x.
The example dataset we'll be using is available at: ftp://share.sph.umich.edu/gotcloud/gotcloudExample.tgz
The example dataset we'll be using is available at: ftp://share.sph.umich.edu/gotcloud/gotcloudExample.tgz
# Create & Change directory to where you want to install the Tutorail data
# Change directory to where you want to install the Tutorial data
# Download the dataset tar from the ftp site
# Download the dataset tar from the ftp site
# Extract the tar
# Extract the tar
cd ~
wget ftp://share.sph.umich.edu/gotcloud/gotcloudExample.tgz # Download
wget ftp://share.sph.umich.edu/gotcloud/gotcloudExample.tgz # Download
tar xvf gotcloudExample.tgz --strip 1 # Extract
tar xvf gotcloudExample.tgz # Extracts into gotcloudExample/
== STEP 2 : Run GotCloud Alignment Pipeline ==
== STEP 2 : Run GotCloud Alignment Pipeline ==
Run the alignment pipeline (the example aligns 2 samples) :
Run the alignment pipeline (the example aligns 2 samples) :
~/gotcloud/gotcloud align --conf ~/gotcloudExample/[[#Alignment Configuration File|GBR2align.conf]] --outdir [[#Alignment Output Directory|~/gotcloudTutorialOut]]
Upon successful completion of the alignment pipeline (about 1-2 minutes), you will see the following message:
Upon successful completion of the alignment pipeline (about 1-2 minutes), you will see the following message:
The final BAM files produced by the alignment pipeline are:
The final BAM files produced by the alignment pipeline are:
ls $GCOUT/bams
ls ~/gotcloudTutorialOut/bams
In this directory you will see:
In this directory you will see:
* BAM (.bam) files - 1 per sample
* BAM (.bam) files - 1 per sample
The Quality Control (QC) files are:
The Quality Control (QC) files are:
ls $GCOUT/QCFiles
ls ~/gotcloudTutorialOut/QCFiles
In this directory you will see:
In this directory you will see:
* VerifyBamID output files:
* VerifyBamID output files:
Run the variant calling pipeline:
Run the variant calling pipeline:
~/gotcloud/gotcloud snpcall --conf [[GBR60vc.conf]] --outdir ~/gotcloudTutorialOut --numjobs 2 --region 20:42900000-43200000
Upon successful completion of the variant calling pipeline (about 3-4 minutes), you will see the following message:
Upon successful completion of the variant calling pipeline (about 3-4 minutes), you will see the following message:
On SNP Call success, the VCF files of interest are:
On SNP Call success, the VCF files of interest are:
ls $GCOUT/vcfs/chr20/chr20.filtered*
ls ~/gotcloudTutorialOut/vcfs/chr20/chr20.filtered*
This gives you the following files:
This gives you the following files:
* chr20.filtered.vcf.gz.tbi - index file for the vcf file
* chr20.filtered.vcf.gz.tbi - index file for the vcf file
Also in the $GCOUT/vcfs/chr20 directory are intermediate files:
Also in the ~/gotcloudTutorialOut/vcfs/chr20 directory are intermediate files:
* the whole chromosome variant calls prior to any filtering:
* the whole chromosome variant calls prior to any filtering:
** chr20.merged.sites.vcf - without per sample genotypes
** chr20.merged.sites.vcf - without per sample genotypes
* 40000001.45000000 subdirectory contains the data for just that region.
* 40000001.45000000 subdirectory contains the data for just that region.
The $GCOUT/split/chr20 folder contains a VCF with just the sites that pass the filters.
The ~/gotcloudTutorialOut/split/chr20 folder contains a VCF with just the sites that pass the filters.
ls $GCOUT/split/chr20/
ls ~/gotcloudTutorialOut/split/chr20/
* '''chr20.filtered.PASS.vcf.gz ''' – vcf of just sites that pass all filters
* '''chr20.filtered.PASS.vcf.gz ''' – vcf of just sites that pass all filters
* chr20.filtered.PASS.split.1.vcf.gz - intermediate file
* chr20.filtered.PASS.split.1.vcf.gz - intermediate file
Run the LD-aware genotype refinement pipeline:
Run the LD-aware genotype refinement pipeline:
~/gotcloud/gotcloud ldrefine --conf [[GBR60vc.conf]] --outdir ~/gotcloudTutorialOut --numjobs 2
Upon successful completion of this pipeline, you will see the following message:
Upon successful completion of this pipeline, you will see the following message:
The output from the beagle step of the genotype refinement pipeline is found in:
The output from the beagle step of the genotype refinement pipeline is found in:
ls $GCOUT/beagle/chr20/chr20.filtered.PASS.beagled.vcf.gz $GCOUT/beagle/chr20/chr20.filtered.PASS.beagled.vcf.gz.tbi
ls ~/gotcloudTutorialOut/beagle/chr20/chr20.filtered.PASS.beagled.vcf.gz ~/gotcloudTutorialOut/beagle/chr20/chr20.filtered.PASS.beagled.vcf.gz.tbi
The output from the thunderVcf (final step) of the genotype refinement pipeline is found in:
The output from the thunderVcf (final step) of the genotype refinement pipeline is found in:
ls $GCOUT/thunder/chr20/GBR/chr20.filtered.PASS.beagled.GBR.thunder.vcf.gz $GCOUT/thunder/chr20/GBR/chr20.filtered.PASS.beagled.GBR.thunder.vcf.gz.tbi
ls ~/gotcloudTutorialOut/thunder/chr20/GBR/chr20.filtered.PASS.beagled.GBR.thunder.vcf.gz ~/gotcloudTutorialOut/thunder/chr20/GBR/chr20.filtered.PASS.beagled.GBR.thunder.vcf.gz.tbi
(If you need to install EPACTS, please refer to the documentation at [[EPACTS#Installation_Details]])
(If you need to install EPACTS, please refer to the documentation at [[EPACTS#Installation_Details]])
$EPACTS/epacts single --vcf $GCOUT/vcfs/chr20/chr20.filtered.vcf.gz --ped $GCDATA/test.GBR60.ped \\
$EPACTS/epacts single --vcf ~/gotcloudTutorialOut/vcfs/chr20/chr20.filtered.vcf.gz --ped ~/gotcloudExample/test.GBR60.ped \\
--out $GCOUT/epacts --test q.linear --run 1 --top 1 --chr 20
--out ~/gotcloudTutorialOut/epacts --test q.linear --run 1 --top 1 --chr 20
Upon successful run, you will see files starting with $GCOUT/epacts
Upon successful run, you will see files starting with ~/gotcloudTutorialOut/epacts
ls $GCOUT/epacts*
ls ~/gotcloudTutorialOut/epacts*
To see the top associated variants, you can run
To see the top associated variants, you can run
less $GCOUT/epacts.epacts.top5000
less ~/gotcloudTutorialOut/epacts.epacts.top5000
To see the locus-zoom like plot, you can type the following command (assuming GNU gnuplot 4.2 or higher version was installed)
To see the locus-zoom like plot, you can type the following command (assuming GNU gnuplot 4.2 or higher version was installed)
xpdf $GCOUT/epacs.zoom.20.42987877.pdf
xpdf ~/gotcloudTutorialOut/epacs.zoom.20.42987877.pdf
Click [[Media:EPACTS TEST.zoom.20.42987877.pdf | Exampe LocusZoom PDF]] to see the expected output pdf
Click [[Media:EPACTS TEST.zoom.20.42987877.pdf | Exampe LocusZoom PDF]] to see the expected output pdf
** AS - assembly value to put in the BAM
** AS - assembly value to put in the BAM
The index file and chromosome 20 references used in this tutorial are included with the example data under the $GCDATA directory. The tutorial uses chromosome 20 only references in order to speed the processing time.
The index file and chromosome 20 references used in this tutorial are included with the example data under the ~/gotcloudExample directory. The tutorial uses chromosome 20 only references in order to speed the processing time.
When running with your own data, you will need to update the:
When running with your own data, you will need to update the:
* INDEL_PREFIX - Indel sites file (need for variant calling pipeline)
* INDEL_PREFIX - Indel sites file (need for variant calling pipeline)
The tutorial configuration file is setup to point to the required chromosome 20 reference files which are included with the tutorial example data in $GCDATA/chr20Ref/.
The tutorial configuration file is setup to point to the required chromosome 20 reference files which are included with the tutorial example data in ~/gotcloudExample/chr20Ref/.
If you are running more than just chromosome 20, you will need whole genome reference files which can be downloaded from [[GotCloudReference]].
If you are running more than just chromosome 20, you will need whole genome reference files which can be downloaded from [[GotCloudReference]].
