GotCloud: Creating a New Pipeline - Revision history

Pjvh: /* Command Line Parameters */ fix formatting about BAM_LIST

2015-08-31T18:55:25Z

Command Line Parameters: fix formatting about BAM_LIST

← Older revision		Revision as of 14:55, 31 August 2015
Line 180:		Line 180:
	* <code>--maxlocaljobs</code> <number> - number of jobs to allow to run when batchtype is local (default 10) - does not validate for commands running LOCAL		* <code>--maxlocaljobs</code> <number> - number of jobs to allow to run when batchtype is local (default 10) - does not validate for commands running LOCAL
	* <code>--region</code> <region to process> - like snpcall, specifies a single region to process		* <code>--region</code> <region to process> - like snpcall, specifies a single region to process
	* <code>--bam_list\|list\|bamlist\|bam_index\|bamindex</code> <bam list file> - overrides		* <code>--bam_list\|list\|bamlist\|bam_index\|bamindex</code> <bam list file> - overrides <code>BAM_LIST</code>, the list of sample bam files to process
	<code>BAM_LIST</code> list of sample~~\tbam~~ to process
	* <code>--out_dir\|outdir</code> <output directory> - overrides <code>OUT_DIR</code>		* <code>--out_dir\|outdir</code> <output directory> - overrides <code>OUT_DIR</code>
	* <code>--batchtype</code> <type> - overrides <code>BATCHTYPE</code>		* <code>--batchtype</code> <type> - overrides <code>BATCHTYPE</code>
Line 198:		Line 197:
	* <code>--keeptmp</code> - overrides <code>KEEP_TMP</code>		* <code>--keeptmp</code> - overrides <code>KEEP_TMP</code>
	* <code>--keeplog</code> - overrides <code>KEEP_LOG</code>		* <code>--keeplog</code> - overrides <code>KEEP_LOG</code>



	== Example Pipelines Created ==		== Example Pipelines Created ==

Pjvh: /* Overall Pipeline Definition */ spelling

2015-08-31T18:28:37Z

Overall Pipeline Definition: spelling

← Older revision		Revision as of 14:28, 31 August 2015
Line 80:		Line 80:
	* NO_CRAM - do not allow CRAM files as input		* NO_CRAM - do not allow CRAM files as input
	* MAKE_BASE_NAME_PIPE - base makefile name		* MAKE_BASE_NAME_PIPE - base makefile name
	* MAKE_OPTS - ~~otpions~~ to pass to the make command that runs the jobs.		* MAKE_OPTS - options to pass to the make command that runs the jobs.
	* BAM_DEPEND - set to TRUE if you want the BAM file to be included as a make dependency		* BAM_DEPEND - set to TRUE if you want the BAM file to be included as a make dependency

Mktrost: /* Example Pipelines Created */

2015-06-30T18:56:30Z

Example Pipelines Created

← Older revision		Revision as of 14:56, 30 June 2015
Line 204:		Line 204:
	Look for sections & <code>STEPS</code> in the defaults.		Look for sections & <code>STEPS</code> in the defaults.
	https://github.com/statgen/gotcloud/blob/master/bin/gotcloudDefaults.conf		https://github.com/statgen/gotcloud/blob/master/bin/gotcloudDefaults.conf
			https://github.com/statgen/gotcloud/blob/alignPrep/bin/gotcloudDefaults.conf

Mktrost: /* Overall Pipeline Definition */

2015-06-30T18:51:55Z

Overall Pipeline Definition

← Older revision		Revision as of 14:51, 30 June 2015
Line 68:		Line 68:
	* BATCH_TYPE		* BATCH_TYPE
	* IGNORE_SM_CHECK - turn off the default validation that the @RG SM tag matches the bam list sample name.		* IGNORE_SM_CHECK - turn off the default validation that the @RG SM tag matches the bam list sample name.
	* IGNORE_REF_CHR_CHECK		* IGNORE_REF_CHR_CHECK - turn off the default validation that checks that all of the BAM's chromosomes are in the reference file - eventually we may update to just validate those in CHRS.
	* OUT_DIR		* OUT_DIR
	* BAM_LIST		* BAM_LIST
Line 88:		Line 88:
	* The BAM_LIST file can contain config values within it - the overall pipeline section will be checked for those config values.		* The BAM_LIST file can contain config values within it - the overall pipeline section will be checked for those config values.
	* By default if a value is not defined in the section, it will check global.		* By default if a value is not defined in the section, it will check global.



	==== Configure Each Step ====		==== Configure Each Step ====

Mktrost: /* Defining a New Pipeline */

2015-06-30T18:50:33Z

Defining a New Pipeline

← Older revision		Revision as of 14:50, 30 June 2015
Line 47:		Line 47:
	# [[#Overall Pipeline Definition\|Overall Pipeline Definition]]		# [[#Overall Pipeline Definition\|Overall Pipeline Definition]]
	#* Basics for the overall pipeline		#* Basics for the overall pipeline
	#* '''NOTE: ~~Curretnly~~, configurations set in the overall pipeline's section do not by default pass onto the step's configurations'''		#* '''NOTE: Currently, configurations set in the overall pipeline's section do not by default pass onto the step's configurations'''
	# [[#Configure Each Step\|Configure Each Step]]		# [[#Configure Each Step\|Configure Each Step]]

Mktrost at 18:50, 30 June 2015

2015-06-30T18:50:18Z

@@ Line 76: / Line 76: @@
 * UNIFORM_TARGET_BED
 * OFFSET_OFF_TARET
-* CHRS
+* CHRS - defines which chromosomes to run.
 * UNIT_CHUNK
 * NO_CRAM - do not allow CRAM files as input
@@ Line 164: / Line 164: @@
 ** be its own Makefile target/.OK file
 ** run independently on the cluster

Mktrost: /* Configure Each Step */

2015-06-30T18:12:30Z

Configure Each Step

← Older revision		Revision as of 14:12, 30 June 2015
Line 92:		Line 92:

	==== Configure Each Step ====		==== Configure Each Step ====
	~~<ol>~~		'''Create a section for each step'''
	~~<li>~~Create a section for each step		* Example: <code>[stepName1]</code>
	<ul><li> ~~Example~~:
	<dd> <~~pre~~>~~[stepName1]~~</~~pre~~>
	<~~/li~~>		====Required keys for each step:====
	</ul>
	<ol>		# <code>DEPEND</code> - dependencies for this step
	<li>~~Set required~~ keys for ~~each~~ step:		#: Valid Values (separate multiple dependencies with a space):
	<ol>		#:*<code>BAM</code>
	<li><code>~~DEPEND~~</code> - ~~dependencies~~ for this step		#:*Name of step that must complete prior to this step
	<dd> ~~Valid Values (separate multiple dependencies~~ with a ~~space~~):		#:*PER_SAMPLE_BAM??? can only be BAM or PER_SAMPLE_BAM
	<ul>		#<code>OUTPUT</code> - name of output file
	<li><code>BAM</code></li>		#* See below for temporary keys for step iteration
	<li>~~Name of~~ step that ~~must complete prior to~~ this step</li>		#<code>CMD</code> - command for running the step
	<li></li>		#* See below for temporary keys for step iteration
	</ul>
	</li>
	<li><code>OUTPUT</code> ~~- name of output file~~		====Optional Step Settings:====
	</li>		General Settings:
	<li><code>~~CMD~~</code> - ~~command for running the step~~		* <code>LOCAL</code> - run the step locally rather than on the cluster
	</li>		* <code>NEED_BAI</code> - Set if a step requires a BAI file
	</ol>		** Per chromosome steps always require a BAI file
	</li>		** Tells GotCloud to fail if a BAI can't be found
	</ol>		* <code>BAM_DEPEND</code> - Add the BAM file as a Makefile dependency for this step
	</li>
	</ol>		Settings to limit which samples this step runs on:
			* <code>SAMPLES</code> - use this to define a step to run only for samples with a single BAM or multiple BAMs (merging)
			*: Possible values:
			: <code>MULTI_BAM</code> - run the step only for samples that have multiple BAMs
			: <code>SINGLE_BAM</code> - run the step only for samples that have one BAM
			*Deprecated settings - still in pipeline.pl and may or may not work:
			** <code>MULTI_ONLY</code> - set to non-blank if step should run if there are more than 1 input per output.
			** <code>SINGLE_ONLY</code> - set to non-blank if step should run if there is only 1 input per output.

			Joining multiple inputs for a single output:
			* Can occur if there are multiple dependencies
			* Can occur if a step runs at a more generic iteration level than a dependency
			* <code>INPUT_JOIN</code> - value to pass to perl "join" command for joining multiple inputs for each output.
			** Looks across all dependencies
			* <code>dependStepName_JOIN</code> - how to join the "dependStepName"'s output into the command line for a step that depends on it if there are multiple outputs per input of this step
			** Substitutes <code>?(${depend}/OUTPUT)</code> with perl "join" using the specified value to join multiple outputs for that dependency

			Log Output filenames
			* <code>FILELIST</code> - writes/appends the iteration's output file name into the specified file list.
			** Typically will be used in a later "merge" step
			** See below for temporary keys for step iteration that can be used in this filename
			*** Temporary keys can be more general than those in OUTPUT, but cannot be more specific.


			====Iterating a command for each Bam/Sample/Chromosome/Region====
			Temporary keys are used when iterating a command per BAM/sample/chromosome/region.
			* Specify using <code>?()</code> rather than <code>$()</code>
			* Temporary keys can be used in:
			** <code>OUTPUT</code>
			** <code>CMD</code>
			** <code>FILELIST</code>
			* They will be substituted as it iterates
			* How to iterate a command is determined by the temporary keys in <code>OUTPUT</code>
			* Temporary Keys for determining iterations:
			** <code>?(BAM)</code> - per BAM per sample
			** <code>?(SAMPLE)</code> - per sample
			** <code>?(CHR)</code> - per chromosome
			** <code>?(START)</code> - Per region of a Chromosome (must also include <code>?(CHR)</code>):
			* Additional Temporary Keys:
			** <code>?(END)</code> - end of the region - only used if <code>?(START)</code> is also specified.
			** <code>?(INPUT)</code>
			** <code>?(${depend}/OUTPUT)</code>

			'''Notes:'''
			* Currently each step iteration will:
			** be its own Makefile target/.OK file
			** run independently on the cluster

Mktrost: /* Defining a New Pipeline */

2015-06-30T17:08:18Z

Defining a New Pipeline

@@ Line 43: / Line 43: @@
 </ul>
 === Defining a New Pipeline ===
 <ol>
 <li> Define a new configuration section for your pipeline
@@ Line 54: / Line 62: @@
 <ul><li> Note: each step must have its own configuration section</li></ul>
 </li>
 <li>Create a section for each step
 <ul><li> Example:

Mktrost at 17:52, 9 March 2015

2015-03-09T17:52:58Z

@@ Line 43: / Line 43: @@
 </ul>
 === Defining a New Pipeline ===
-# Define a new configuration section for your pipeline
+<ol>
-#: <code>[pipelineName]</code>
+<li> Define a new configuration section for your pipeline

Mktrost: /* Creating a New BAM Processing Pipeline */

2015-03-03T20:39:39Z

Creating a New BAM Processing Pipeline

@@ Line 1: / Line 1: @@
 == Creating a New BAM Processing Pipeline ==
 GotCloud allows you to configure new basic BAM processing pipelines via configuration.
 # Define a new configuration section for your pipeline
 #: <code>[pipelineName]</code>