Line 21: |
Line 21: |
| 6. Make a new folder, where you'll run your jobs from. | | 6. Make a new folder, where you'll run your jobs from. |
| | | |
− | 7. Make an email address to send your jobs' status to. I recommend that you don't use your primary one. | + | 7. Make an email address to send your jobs' status to. It'll be hit by hundreds or thousands of emails, so I recommend that you don't use your main email address here. |
| + | :* If you're in a hurry to finish your pipeline, you can find an email address that will text the emails to your phone. Only use that in the second script, though! |
| | | |
| 8. Figure out the name of the Flux account that you're going to use. You can see which Flux accounts you have access to by running <code>mdiag -u $USER</code> and looking at the list after <code>ALIST</code>. | | 8. Figure out the name of the Flux account that you're going to use. You can see which Flux accounts you have access to by running <code>mdiag -u $USER</code> and looking at the list after <code>ALIST</code>. |
| + | :* Eg, <code>sph_flux</code> |
| | | |
| 9. Figure out how many processors you're going to use at once. Run <code>mdiag -a YOU_FLUX_ACCOUNT</code>. I recommend running <code>MAXPROC</code> + <code>MAXIJOB[USER]</code> many jobs. <code>MAXPROC</code> is the number of processors on your account, and <code>MAXIJOB[USER]</code> is the number of jobs that can sit idle in the queue waiting to be run (often 20). | | 9. Figure out how many processors you're going to use at once. Run <code>mdiag -a YOU_FLUX_ACCOUNT</code>. I recommend running <code>MAXPROC</code> + <code>MAXIJOB[USER]</code> many jobs. <code>MAXPROC</code> is the number of processors on your account, and <code>MAXIJOB[USER]</code> is the number of jobs that can sit idle in the queue waiting to be run (often 20). |
| + | :* This number will usually be between 20 and 1000. |
| | | |
− | 10. Figure out which pipelines to run first. They will run in the order glfN, vcfN, pvcfN, filtN, svmN, splitN, allN where N is the name of a chromosome (ie, 1-22 and maybe X and Y). If you skip a step, the <code>make</code> will run it for you. If you're confident you can go straight to the step allN. If you want to babysit the jobs, do them one-at-a-time. Your MAKEFILE_TARGETS will need one for each chromosome, like <code>glf1 glf2 glf3 glf4 glf5 glf6 glf7 glf8 glf9 glf10 glf11 glf12 glf13 glf14 glf15 glf16 glf17 glf18 glf19 glf20 glf21 glf22 </code>. Feel free to use the script <code>perl -e 'print "glf$_ " for 1..22'</code> to mitigate repetitive strain injuries. | + | 10. Figure out which steps to run first. The steps go in the order glfN, vcfN, pvcfN, filtN, svmN, splitN, allN where N is the name of a chromosome (ie, 1-22 and maybe X and Y). If you skip a step, it's not a problem, because <code>make</code> will run it for you. If you're confident that everything will work beautifully, you can go straight to the step <code>allN</code> (or just <code>all</code> as a shortcut). |
| + | :* For example, I used <code>glf1 glf2 glf3 glf4 glf5 glf6 glf7 glf8 glf9 glf10 glf11 glf12 glf13 glf14 glf15 glf16 glf17 glf18 glf19 glf20 glf21 glf22 </code> the first time I ran on Flux. Then I ran <code>vcf1 vcf2...<code>, and on down the list until finally <code>all</code>. |
| + | :* Feel free to use the script <code>perl -e 'print "glf$_ " for 1..22'</code> to mitigate repetitive strain injuries. |
| | | |
| 11. Inside that new folder, make a new file named <code>pbs.options</code> that contains the following: | | 11. Inside that new folder, make a new file named <code>pbs.options</code> that contains the following: |
Line 40: |
Line 45: |
| #PBS -j oe | | #PBS -j oe |
| | | |
− | 12. Still inside that folder, create a script that you will submit to flux. It should look like this: | + | 12. Still inside that folder, create a script that you will submit to flux. Let's name it <code>script_thats_in_charge.sh</code>. It should look like this: |
| #!/bin/sh | | #!/bin/sh |
| | | |
Line 57: |
Line 62: |
| | | |
| echo "job ended with status $? at $(date)" | | echo "job ended with status $? at $(date)" |
| + | |
| + | 13. Run <code>qsub script_thats_in_charge.sh</code>. It's important that you run this in the same folder where <code>pbs.options</code> lives. |
| + | |
| + | 14. Once that finishes, if any steps remain, then update YOUR_MAKEFILE_TARGETS_FROM_STEP_10 and go back to step 13. |