Line 7: |
Line 7: |
| These notes assume you have already created an EC2 account and have the certificates and keys set up properly. | | These notes assume you have already created an EC2 account and have the certificates and keys set up properly. |
| | | |
− | == Launch an instance == | + | |
| + | == Create new GotCloud AMI from StarCluster AMI == |
| + | === Launch an instance === |
| | | |
| <code> | | <code> |
Line 21: |
Line 23: |
| ## Enter in the search box: <code>starcluster-base-ubuntu</code> | | ## Enter in the search box: <code>starcluster-base-ubuntu</code> |
| ## Select: <code>starcluster-base-ubuntu-12.04-x86_64 - ami-765b3e1f</code> | | ## Select: <code>starcluster-base-ubuntu-12.04-x86_64 - ami-765b3e1f</code> |
− | # Select the Instance Type: <code>Micro t1.micro</code> | + | # Select the Instance Type: <code>Compute optimized c3.2xlarge</code> |
− | #* Since we are just using this instance to setup the system, pick the smallest/cheapest machine. | + | #* You can use a smaller/cheaper machine - I originaly used t1.micro, but I found things go so much faster with a larger machine. |
| # Click: <code>Review and Launch</code> | | # Click: <code>Review and Launch</code> |
| ## Select: <code>Make General Purpose (SSD) the boot volume for this instance.</code> | | ## Select: <code>Make General Purpose (SSD) the boot volume for this instance.</code> |
Line 34: |
Line 36: |
| # Select the key/pair you want to use & Launch | | # Select the key/pair you want to use & Launch |
| | | |
− | == Setup the instance with GotCloud == | + | === Setup the instance with GotCloud === |
| This assumes you have already logged onto the instance. | | This assumes you have already logged onto the instance. |
| | | |
| # Get the latest version of GotCloud: | | # Get the latest version of GotCloud: |
| #* Multiples ways to do this, one way is to do: | | #* Multiples ways to do this, one way is to do: |
− | #*# <code>git clone https://github.com/statgen/gotcloud.git gc_src</code> | + | #*# <code>sudo git clone https://github.com/statgen/gotcloud.git</code> |
| # Download cmake (required to build premo) | | # Download cmake (required to build premo) |
| #*<code>sudo apt-get update</code> | | #*<code>sudo apt-get update</code> |
Line 45: |
Line 47: |
| #*<code>sudo apt-get install cmake</code> | | #*<code>sudo apt-get install cmake</code> |
| ## Build the source (if you obtained the source code). | | ## Build the source (if you obtained the source code). |
− | ### <code>cd gc_src/src</code> | + | ### <code>cd gotcloud/src</code> |
− | ### <code>make</code> | + | ### <code>sudo make</code> |
− | ### <code>cd ..</code> | + | ###* Specify <code>-j #</code> based on the number of CPUs your instance has, if more than 1 |
− | ## Generate the installation.
| |
− | ### <code>cd gc_src/</code>
| |
− | ### <code>./debian/makedeb.sh bin =</code>
| |
| ### <code>cd</code> | | ### <code>cd</code> |
− | # Install: <code>sudo dpkg -i gc_src/gotcloud-bin_1.14.3_amd64.deb</code>
| |
− | # Move installation to home so will be on all nodes: <code>sudo mv /usr/local/gotcloud .</code>
| |
− | # Remove source: <code>rm -rf gc_src/</code> (required since not enough storage space.)
| |
| # Get the reference files | | # Get the reference files |
| ## wget ftp://anonymous@share.sph.umich.edu/gotcloud/ref/h37-db135-v3.tgz | | ## wget ftp://anonymous@share.sph.umich.edu/gotcloud/ref/h37-db135-v3.tgz |
Line 60: |
Line 56: |
| # Move reference to gotcloud directory: <code>sudo mv gotcloud.ref gotcloud</code> | | # Move reference to gotcloud directory: <code>sudo mv gotcloud.ref gotcloud</code> |
| # Remove tar file: <code>rm h37-db135-v3.tgz</code> | | # Remove tar file: <code>rm h37-db135-v3.tgz</code> |
− | # Add aliases for GotCloud and tabix: vi .bash_aliases | + | # Set the paths, by updating .profile: <code>vi .profile</code> |
− | ## <code>alias gotcloud='/home/ubuntu/gotcloud/bin/gotcloud'</code> | + | #* <code>i</code> |
− | ## <code>alias tabix='/home/ubuntu/gotcloud/bin/tabix'</code> | + | #: <pre>if [ -d "$HOME/gotcloud" ] ; then PATH="$HOME/gotcloud:$PATH" fi if [ -d "$HOME/gotcloud/bin" ] ; then PATH="$HOME/gotcloud/bin:$PATH" fi if [ -d "$HOME/gotcloud/scripts" ] ; then PATH="$HOME/gotcloud/scripts:$PATH" fi</pre> |
− | | + | #* <code>ESC</code> |
− | | + | #* <code>:q</code> |
− | == Create Image ==
| |
| | | |
− | == Set Up Swap Space == | + | === Set Up Swap Space === |
| | | |
| Issue the command '''swapon -s''' to see if there is swap space. | | Issue the command '''swapon -s''' to see if there is swap space. |
Line 87: |
Line 82: |
| </code> | | </code> |
| | | |
− | == Install the Software == | + | === Cleanup the instance for creating an AMI === |
| + | # Go to : [[#Cleanup Instance for AMI Creation|Cleanup Instance for AMI Creation]] |
| + | |
| + | === Create the AMI === |
| + | # Go to : [[#Create the AMI|Create the AMI]] |
| + | |
| + | |
| + | == Update the GotCloud AMI == |
| + | # Start an instance of the current GotCloud AMI |
| + | #* Suggest an instance with some CPU so you can parallelize the "make" call. |
| + | # Login as ubuntu |
| + | # <code>cd gotcloud</code> |
| + | # <code>sudo git pull</code> |
| + | # <code>cd gotcloud/src</code> |
| + | # <code>sudo make</code> |
| + | #* Specify <code>-j #</code> based on the number of CPUs your instance has |
| + | # <code>cd</code> |
| + | # Go to : [[#Create the AMI|Create the AMI]] |
| + | |
| + | |
| + | ==Cleanup Instance for AMI Creation== |
| + | First time from generic/starcluster AMI |
| + | # Disable password-based logins for root |
| + | ## Open /etc/ssh/sshd_config |
| + | ## Change <code>PermitRootLogin yes</code> to <code>PermitRootLogin without-password</code> |
| + | # Disable root access |
| + | ## <code> sudo passwd -l root</code> |
| + | |
| + | |
| + | Each time we generate a new AMI, run: |
| + | <pre>sudo shred -u /etc/ssh/*_key /etc/ssh/*_key.pub |
| + | sudo find / -name "authorized_keys" -exec rm -f {} \; |
| + | rm -rf ~/.ssh |
| + | shred -u ~/.*history |
| + | sudo find /root/.*history /home/*/.*history -exec rm -f {} \; |
| + | history -w |
| + | history -c |
| + | </pre> |
| + | These commands do the following: |
| + | # Remove SSH host key pairs |
| + | # Remove SSH authorized keys |
| + | # Remove ssh |
| + | # Delete shell history |
| + | |
| + | == Create the AMI == |
| + | |
| + | Once your instance is all ready with everything you want, create the AMI. |
| + | |
| + | In your browser at the EC2 Management Console do the following: |
| + | # Select the running instance |
| + | # Right click, <code>Create Image</code> |
| + | # Enter name & Description |
| + | # Ensure volume size is correct |
| + | # Mark delete on terminate |
| + | #:This will take several minutes to complete. |
| + | #:In the EC2 Dashboard, you can monitor the progress. |
| + | #:When it is done, you'll see a new AMI under the list of AMIs. |
| + | # When completed, terminate your old instance |
| + | |
| + | |
| + | == Older/Additional Instructions == |
| + | === Install the Software === |
| | | |
| '''(1)''' There are a number of additional Debian packages that you may well need, so we make | | '''(1)''' There are a number of additional Debian packages that you may well need, so we make |
Line 166: |
Line 222: |
| Run the tests to be sure everything is OK. | | Run the tests to be sure everything is OK. |
| | | |
− | == Configure the Host to be Usable == | + | === Configure the Host to be Usable === |
| | | |
| It is useful to configure /etc/rc.local to do most things you need at boot time. | | It is useful to configure /etc/rc.local to do most things you need at boot time. |
Line 250: |
Line 306: |
| </code> | | </code> |
| | | |
− | == Create the AMI == | + | === Test the new AMI === |
− | | |
− | Once your instance is all ready with the files you want, swap space etc, then create the AMI.
| |
− | In your browser at the EC2 Management Console do the following:
| |
− | | |
− | <code>
| |
− | Create Image
| |
− | Image Name GotCLoud 1.06
| |
− | Image Description: From CSG at University of Michigan
| |
− | Volume Size: 30GB
| |
− | Take defaults otherwise
| |
− | </code>
| |
− | | |
− | This will take several minutes to complete.
| |
− | In the EC2 Dashboard, you can monitor the progress.
| |
− | When it is done, you'll see a new AMI under the list of AMIs.
| |
− | | |
− | Your new AMI should look pretty much like this:
| |
− | | |
− | <code>
| |
− | AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
| |
− | Name: Ubuntu Server 12.04.1 LTS
| |
− | Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
| |
− | Number of Instances: 1
| |
− | Availability Zone: No Preference
| |
− | Instance Type: Micro (t1.micro)
| |
− | Instance Class: On Demand Edit Instance Details
| |
− | EBS-Optimized: No
| |
− | Monitoring: Disabled Termination Protection: Disabled
| |
− | Tenancy: Default
| |
− | Kernel ID: Use Default Shutdown Behavior: Stop
| |
− | RAM Disk ID: Use Default
| |
− | Network Interfaces:
| |
− | Secondary IP Addresses:
| |
− | User Data:
| |
− | IAM Role: Edit Advanced Details
| |
− | Key Pair Name: CSG Edit Key Pair
| |
− | Security Group(s): sg-a098e9c8 Edit Firewall
| |
− | </code>
| |
− | | |
− | == Test the new AMI ==
| |
| | | |
| Launch a new AMI instance and check that files are in the correct places. | | Launch a new AMI instance and check that files are in the correct places. |