Changes

From Genome Analysis Wiki
Jump to navigationJump to search

Creating an AMI on EC2 (view source)

Revision as of 14:54, 14 October 2014

820 bytes added ,  14:54, 14 October 2014
→‎Cleanup Instance for AMI Creation
Line 7: Line 7:  
These notes assume you have already created an EC2 account and have the certificates and keys set up properly.
 
These notes assume you have already created an EC2 account and have the certificates and keys set up properly.
   −
== Launch an instance ==
+
 
 +
== Create new GotCloud AMI from StarCluster AMI ==
 +
=== Launch an instance ===
    
<code>
 
<code>
Line 21: Line 23:  
## Enter in the search box: <code>starcluster-base-ubuntu</code>
 
## Enter in the search box: <code>starcluster-base-ubuntu</code>
 
## Select: <code>starcluster-base-ubuntu-12.04-x86_64 - ami-765b3e1f</code>
 
## Select: <code>starcluster-base-ubuntu-12.04-x86_64 - ami-765b3e1f</code>
−
# Select the Instance Type: <code>Micro t1.micro</code>
+
# Select the Instance Type: <code>Compute optimized c3.2xlarge</code>
−
#* Since we are just using this instance to setup the system, pick the smallest/cheapest machine.
+
#* You can use a smaller/cheaper machine - I originaly used t1.micro, but I found things go so much faster with a larger machine.
 
# Click: <code>Review and Launch</code>
 
# Click: <code>Review and Launch</code>
 
## Select: <code>Make General Purpose (SSD) the boot volume for this instance.</code>
 
## Select: <code>Make General Purpose (SSD) the boot volume for this instance.</code>
Line 34: Line 36:  
# Select the key/pair you want to use & Launch
 
# Select the key/pair you want to use & Launch
   −
== Setup the instance with GotCloud ==
+
=== Setup the instance with GotCloud ===
 
This assumes you have already logged onto the instance.
 
This assumes you have already logged onto the instance.
    
# Get the latest version of GotCloud:
 
# Get the latest version of GotCloud:
 
#* Multiples ways to do this, one way is to do:
 
#* Multiples ways to do this, one way is to do:
−
#*# <code>git clone https://github.com/statgen/gotcloud.git gc_src</code>
+
#*# <code>sudo git clone https://github.com/statgen/gotcloud.git</code>
 
# Download cmake (required to build premo)
 
# Download cmake (required to build premo)
 
#*<code>sudo apt-get update</code>
 
#*<code>sudo apt-get update</code>
Line 45: Line 47:  
#*<code>sudo apt-get install cmake</code>
 
#*<code>sudo apt-get install cmake</code>
 
## Build the source (if you obtained the source code).
 
## Build the source (if you obtained the source code).
−
### <code>cd gc_src/src</code>
+
### <code>cd gotcloud/src</code>
−
### <code>make</code>
+
### <code>sudo make</code>
−
### <code>cd ..</code>
+
###* Specify <code>-j #</code> based on the number of CPUs your instance has, if more than 1
−
## Generate the installation.
  −
### <code>cd gc_src/</code>
  −
### <code>./debian/makedeb.sh bin =</code>
   
### <code>cd</code>
 
### <code>cd</code>
−
# Install: <code>sudo dpkg -i gc_src/gotcloud-bin_1.14.3_amd64.deb</code>
  −
# Move installation to home so will be on all nodes: <code>sudo mv /usr/local/gotcloud .</code>
  −
# Remove source: <code>rm -rf gc_src/</code> (required since not enough storage space.)
   
# Get the reference files
 
# Get the reference files
 
## wget ftp://anonymous@share.sph.umich.edu/gotcloud/ref/h37-db135-v3.tgz
 
## wget ftp://anonymous@share.sph.umich.edu/gotcloud/ref/h37-db135-v3.tgz
 
# Untar: <code>tar xvf h37-db135-v3.tgz</code>
 
# Untar: <code>tar xvf h37-db135-v3.tgz</code>
 +
# Move reference to gotcloud directory: <code>sudo mv gotcloud.ref gotcloud</code>
 +
# Remove tar file: <code>rm h37-db135-v3.tgz</code>
 +
# Set the paths, by updating .profile: <code>vi .profile</code>
 +
#* <code>i</code>
 +
#: <pre>if [ -d "$HOME/gotcloud" ] ; then&#10;    PATH="$HOME/gotcloud:$PATH"&#10;fi&#10;if [ -d "$HOME/gotcloud/bin" ] ; then&#10;    PATH="$HOME/gotcloud/bin:$PATH"&#10;fi&#10;if [ -d "$HOME/gotcloud/scripts" ] ; then&#10;    PATH="$HOME/gotcloud/scripts:$PATH"&#10;fi</pre>
 +
#* <code>ESC</code>
 +
#* <code>:q</code>
   −
 
+
=== Set Up Swap Space ===
−
 
  −
=== Install GotCloud in ubuntu home directory ===
  −
 
  −
Set aliases in .bashrc
  −
 
  −
== Create Image ==
  −
 
  −
== Set Up Swap Space ==
      
Issue the command '''swapon -s''' to see if there is swap space.
 
Issue the command '''swapon -s''' to see if there is swap space.
Line 87: Line 82:  
</code>
 
</code>
   −
== Install the Software ==
+
=== Cleanup the instance for creating an AMI ===
 +
# Go to : [[#Cleanup Instance for AMI Creation|Cleanup Instance for AMI Creation]]
 +
 
 +
=== Create the AMI ===
 +
# Go to : [[#Create the AMI|Create the AMI]]
 +
 
 +
 
 +
== Update the GotCloud AMI ==
 +
# Start an instance of the current GotCloud AMI
 +
#* Suggest an instance with some CPU so you can parallelize the "make" call.
 +
# Login as ubuntu
 +
# <code>cd gotcloud</code>
 +
# <code>sudo git pull</code>
 +
# <code>cd gotcloud/src</code>
 +
# <code>sudo make</code>
 +
#* Specify <code>-j #</code> based on the number of CPUs your instance has
 +
# <code>cd</code>
 +
# Go to : [[#Create the AMI|Create the AMI]]
 +
 
 +
 
 +
==Cleanup Instance for AMI Creation==
 +
First time from generic/starcluster AMI
 +
# Disable password-based logins for root
 +
## Open /etc/ssh/sshd_config
 +
## Change <code>PermitRootLogin yes</code> to <code>PermitRootLogin without-password</code>
 +
# Disable root access
 +
## <code> sudo passwd -l root</code>
 +
 
 +
 
 +
Each time we generate a new AMI, run:
 +
<pre>sudo shred -u /etc/ssh/*_key /etc/ssh/*_key.pub
 +
sudo find / -name "authorized_keys" -exec rm -f {} \;
 +
rm -rf ~/.ssh
 +
shred -u ~/.*history
 +
sudo find /root/.*history /home/*/.*history -exec rm -f {} \;
 +
history -w
 +
history -c
 +
</pre>
 +
These commands do the following:
 +
# Remove SSH host key pairs
 +
# Remove SSH authorized keys
 +
# Remove ssh
 +
# Delete shell history
 +
 
 +
== Create the AMI ==
 +
 
 +
Once your instance is all ready with everything you want, create the AMI.
 +
 
 +
In your browser at the EC2 Management Console do the following:
 +
# Select the running instance
 +
# Right click, <code>Create Image</code>
 +
# Enter name & Description
 +
# Ensure volume size is correct
 +
# Mark delete on terminate
 +
#:This will take several minutes to complete.
 +
#:In the EC2 Dashboard, you can monitor the progress.
 +
#:When it is done, you'll see a new AMI under the list of AMIs.
 +
# When completed, terminate your old instance
 +
 
 +
 
 +
== Older/Additional Instructions ==
 +
=== Install the Software ===
    
'''(1)''' There are a number of additional Debian packages that you may well need, so we make
 
'''(1)''' There are a number of additional Debian packages that you may well need, so we make
Line 166: Line 222:  
Run the tests to be sure everything is OK.
 
Run the tests to be sure everything is OK.
   −
== Configure the Host to be Usable ==
+
=== Configure the Host to be Usable ===
    
It is useful to configure /etc/rc.local to do most things you need at boot time.
 
It is useful to configure /etc/rc.local to do most things you need at boot time.
Line 250: Line 306:  
</code>
 
</code>
   −
== Create the AMI ==
+
=== Test the new AMI ===
−
 
  −
Once your instance is all ready with the files you want, swap space etc, then create the AMI.
  −
In your browser at the EC2 Management Console do the following:
  −
 
  −
<code>
  −
  Create Image
  −
    Image Name  GotCLoud 1.06
  −
    Image Description:  From CSG at University of Michigan
  −
    Volume Size:  30GB
  −
    Take defaults otherwise
  −
</code>
  −
 
  −
This will take several minutes to complete.
  −
In the EC2 Dashboard, you can monitor the progress.
  −
When it is done, you'll see a new AMI under the list of AMIs.
  −
 
  −
Your new AMI should look pretty much like this:
  −
 
  −
<code>
  −
  AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
  −
  Name: Ubuntu Server 12.04.1 LTS
  −
  Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
  −
  Number of Instances: 1
  −
  Availability Zone: No Preference
  −
  Instance Type: Micro (t1.micro)
  −
  Instance Class: On Demand Edit Instance Details
  −
  EBS-Optimized: No
  −
  Monitoring: Disabled Termination Protection: Disabled
  −
  Tenancy: Default
  −
  Kernel ID: Use Default Shutdown Behavior: Stop
  −
  RAM Disk ID: Use Default
  −
  Network Interfaces:
  −
  Secondary IP Addresses:
  −
  User Data:
  −
  IAM Role: Edit Advanced Details
  −
  Key Pair Name: CSG Edit Key Pair
  −
  Security Group(s): sg-a098e9c8 Edit Firewall
  −
</code>
  −
 
  −
== Test the new AMI ==
      
Launch a new AMI instance and check that files are in the correct places.
 
Launch a new AMI instance and check that files are in the correct places.
Retrieved from "http://genome.sph.umich.edu/wiki/Special:MobileDiff/11699...11713"

Navigation menu