Difference between revisions of "Creating an AMI on EC2"
From Genome Analysis Wiki
Jump to navigationJump to searchTerry Gliedt (talk | contribs) |
Terry Gliedt (talk | contribs) |
||
| Line 140: | Line 140: | ||
<code> | <code> | ||
ubuntu@ip-10-254-60-210:~$ sudo more /etc/rc.local | ubuntu@ip-10-254-60-210:~$ sudo more /etc/rc.local | ||
| − | #!/bin/sh | + | #!/bin/sh |
| − | # | + | # |
| − | # rc.local | + | # rc.local |
| − | # | + | # |
| − | # This script is executed at the end of each multiuser runlevel. | + | # This script is executed at the end of each multiuser runlevel. |
| − | # Make sure that the script will "exit 0" on success or any other | + | # Make sure that the script will "exit 0" on success or any other |
| − | # value on error. | + | # value on error. |
| − | # | + | # |
| − | # In order to enable or disable this script just change the execution | + | # In order to enable or disable this script just change the execution |
| − | # bits. | + | # bits. |
| − | # | + | # |
| − | # By default this script does nothing. | + | # By default this script does nothing. |
| − | USER=ubuntu | + | USER=ubuntu |
| − | THOUSANDG=/mnt/1000g | + | THOUSANDG=/mnt/1000g |
| − | FILES3=/etc/passwd-s3fs # Where s3fs access info will live | + | FILES3=/etc/passwd-s3fs # Where s3fs access info will live |
| − | S3ERR=/tmp/s3fs.err | + | S3ERR=/tmp/s3fs.err |
| − | # These are needed for s3fs access | + | # These are needed for s3fs access |
| − | AWSACCESSKEYID=AKIAxxxxxxZ3YCZF2Q | + | AWSACCESSKEYID=AKIAxxxxxxZ3YCZF2Q |
| − | AWSSECRETACCESSKEY=ft1eJa3WxxxxxxxNlbA08x/G8iMqkMIkJjFCIGf | + | AWSSECRETACCESSKEY=ft1eJa3WxxxxxxxNlbA08x/G8iMqkMIkJjFCIGf |
| − | + | ||
| − | + | ||
| − | # Check that we have swap set up | + | # Check that we have swap set up |
| − | a=`swapon -s | grep -v File` | + | a=`swapon -s | grep -v File` |
| − | if [ "$a" = "" ]; then | + | if [ "$a" = "" ]; then |
| − | + | echo "#######################################################" | |
| − | + | echo "# You have no SWAP file set up" | |
| − | + | echo "" | |
| − | + | echo "# swap=/mnt/swapfile" | |
| − | + | echo "# sudo dd if=/dev/zero of=$swap bs=1073741824 count=20" | |
| − | + | echo "# sudo chown root:root $swap" | |
| − | + | echo "# sudo mkswap $swap" | |
| − | + | echo "# sudo chmod 0600 $swap" | |
| − | + | echo "# sudo swapon $swap" | |
| − | + | echo "" | |
| − | + | echo "# If need be, add to /etc/fstab" | |
| − | + | echo "# echo "$swap none swap sw 0 0" >> /etc/fstab" | |
| − | + | echo "#######################################################" | |
| − | fi | + | fi |
| − | + | ||
| − | # Set up for GotCloud | + | # Set up for GotCloud |
| − | gc=/gotcloud.mnt | + | gc=/gotcloud.mnt |
| − | if [ ! -r $gc/release_version.txt ]; then | + | if [ ! -r $gc/release_version.txt ]; then |
| − | + | mkdir -p $gc | |
| − | + | mount /dev/xvdg $gc | |
| − | + | if [ -d $gc/gotcloud.ref ]; then | |
| − | + | echo "#######################################################" | |
| − | + | echo "# GotCloud is set up on $gc" | |
| − | + | echo "#######################################################" | |
| − | + | fi | |
| − | fi | + | fi |
| − | + | ||
| − | # Set up access to S3 storage as normal filesystem | + | # Set up access to S3 storage as normal filesystem |
| − | echo "${AWSACCESSKEYID}:$AWSSECRETACCESSKEY" > $FILES3 | + | echo "${AWSACCESSKEYID}:$AWSSECRETACCESSKEY" > $FILES3 |
| − | chown root.root $FILES3 | + | chown root.root $FILES3 |
| − | chmod 640 $FILES3 | + | chmod 640 $FILES3 |
| − | + | ||
| − | usermod -aG fuse $USER | + | usermod -aG fuse $USER |
| − | + | ||
| − | # Setup 1000genomes | + | # Setup 1000genomes |
| − | mkdir -p $THOUSANDG | + | mkdir -p $THOUSANDG |
| − | if [ ! -r $THOUSANDG/release ]; then | + | if [ ! -r $THOUSANDG/release ]; then |
| − | + | chown $USER.$USER $THOUSANDG | |
| − | + | /usr/local/bin/s3fs -o allow_other 1000genomes $THOUSANDG > $S3ERR 2>&1 | |
| − | + | if [ ! -r $THOUSANDG/alignment.index ]; then | |
| − | + | echo "#######################################################" | |
| − | + | echo "# 1000genomes is not set up on $THOUSANDG" | |
| − | + | echo "# See S3FS errors in $S3ERR" | |
| − | + | echo "#######################################################" | |
| − | + | fi | |
| − | + | df -h | |
| − | fi | + | fi |
| − | exit 0 | + | exit 0 |
| − | |||
</code> | </code> | ||
Revision as of 13:10, 17 June 2013
Notes About Creating a New EC2 AMI
Back to parent: GotCloud
The following are notes taken when creating the Amazon Machine Instance used for the CSG pipeline process. These notes assume you have already created an EC2 account and have the certificates and keys set up properly.
Launch an instance
Login to https://console.aws.amazon.com/ec2 # EC2 Management Console
Pay attention to the region you are using, at least for now it seems any StarCluster activity must be in us-east-1. Launch a new instance which we will use to set up the software and ultimately save it as an AMI.
EC2 DashBoard -> Launch Instance
Class Wizard
Ubuntu Server 12.04.1 LTS 64 bit
Instance type -> Micro, EC2, no preference # Memory size does not matter
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G -> Save -> Continue # Storage size does not matter
Key Name = GotCloud 1.06a
Create Key/Pair if you need to, Name the PEM and save the pem file for access by ssh
Choose a Security Group (take default)
Launch
No need to Create Status Check Alarms
No need to Create EBS Volumes
Set Up Swap Space
Issue the command swapon -s to see if there is swap space. If there is only a header line, you need to add a swap file like this:
df -h # Be sure there's enough space, decide on swap size
# Create a file /swap to use (assuming / is large enough)
sudo bash # Run these commands as root
swap=/swap
dd if=/dev/zero of=$swap bs=524288 count=16384 # 8GB swap on t1.micro 15G=bs=1073741824 count=15
chown root:root $swap
mkswap $swap
chmod 0600 $swap
swapon $swap
echo "$swap none swap sw 0 0" >> /etc/fstab
swapon -s # Should show the swap device
Install the Software
(1) There are a number of additional Debian packages that you may well need, so we make sure they are all installed.
sudo apt-get update
sudo apt-get upgrade # Apply maintenance
sudo apt-get install java-common default-jre make libssl0.9.8
sudo apt-get install libnet-amazon-ec2-perl s3cmd
sudo apt-get install make g++ libcurl4-openssl-dev libssl-dev libxml2-dev libfuse-dev
(2) S3fs allows one to access S3 storage as a conventional file system. This can be quite handy, if it is set up properly. Our recent experience is that the 1000 Genomes data is has many files with incorrect permissions. Still if you're lucky, your data will be useful. Install the software like this:
mkdir -p ~/src
cd ~/src
wget http://s3fs.googlecode.com/files/s3fs-1.68.tar.gz
tar xzvf s3fs-1.68.tar.gz
cd s3fs*
./configure
sudo make install
(3) Configure s3cmd. This will ask for your AWS ID and Secret Key. If creates a file in ~/.s3cfg
s3cmd --configure
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3
Access Key: AKI1234QEUWZ3YCZF2Q
Secret Key: ft1eJa1234NE8iitNlbA08x/G8iMqkMI1234IGf
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: password_you_do_not_need_to_know
Path to GPG program [/usr/bin/gpg]:
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]:
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't conect to S3 directly
HTTP Proxy server name:
New settings:
Access Key: AKI1234QEUWZ3YCZF2Q
Secret Key: ft1eJa1234NE8iitNlbA08x/G8iMqkMI1234IGf
Encryption password: password_you_do_not_need_to_know
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name:
HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n]
Please wait...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)
Save settings? [y/N] y
Configuration saved to '/home/ubuntu/.s3cfg'
(4) Follow the instructions to install the GotCloud Debian packages Run the tests to be sure everything is OK.
Configure the Host to be Usable
It is useful to configure /etc/rc.local to do most things you need at boot time. There are many other ways to do this, but here's one simple way - create the file /etc/rc.local (as root). The following example sets up access details for s3cmd and s3fs (use your own credentials).
ubuntu@ip-10-254-60-210:~$ sudo more /etc/rc.local
#!/bin/sh
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
USER=ubuntu
THOUSANDG=/mnt/1000g
FILES3=/etc/passwd-s3fs # Where s3fs access info will live
S3ERR=/tmp/s3fs.err
# These are needed for s3fs access
AWSACCESSKEYID=AKIAxxxxxxZ3YCZF2Q
AWSSECRETACCESSKEY=ft1eJa3WxxxxxxxNlbA08x/G8iMqkMIkJjFCIGf
# Check that we have swap set up
a=`swapon -s | grep -v File`
if [ "$a" = "" ]; then
echo "#######################################################"
echo "# You have no SWAP file set up"
echo ""
echo "# swap=/mnt/swapfile"
echo "# sudo dd if=/dev/zero of=$swap bs=1073741824 count=20"
echo "# sudo chown root:root $swap"
echo "# sudo mkswap $swap"
echo "# sudo chmod 0600 $swap"
echo "# sudo swapon $swap"
echo ""
echo "# If need be, add to /etc/fstab"
echo "# echo "$swap none swap sw 0 0" >> /etc/fstab"
echo "#######################################################"
fi
# Set up for GotCloud
gc=/gotcloud.mnt
if [ ! -r $gc/release_version.txt ]; then
mkdir -p $gc
mount /dev/xvdg $gc
if [ -d $gc/gotcloud.ref ]; then
echo "#######################################################"
echo "# GotCloud is set up on $gc"
echo "#######################################################"
fi
fi
# Set up access to S3 storage as normal filesystem
echo "${AWSACCESSKEYID}:$AWSSECRETACCESSKEY" > $FILES3
chown root.root $FILES3
chmod 640 $FILES3
usermod -aG fuse $USER
# Setup 1000genomes
mkdir -p $THOUSANDG
if [ ! -r $THOUSANDG/release ]; then
chown $USER.$USER $THOUSANDG
/usr/local/bin/s3fs -o allow_other 1000genomes $THOUSANDG > $S3ERR 2>&1
if [ ! -r $THOUSANDG/alignment.index ]; then
echo "#######################################################"
echo "# 1000genomes is not set up on $THOUSANDG"
echo "# See S3FS errors in $S3ERR"
echo "#######################################################"
fi
df -h
fi
exit 0
Create the AMI
Once your instance is all ready with the files you want, swap space etc, then create the AMI. In your browser at the EC2 Management Console do the following:
Create Image
Image Name GotCLoud 1.06
Image Description: From CSG at University of Michigan
Volume Size: 30GB
Take defaults otherwise
This will take several minutes to complete. In the EC2 Dashboard, you can monitor the progress. When it is done, you'll see a new AMI under the list of AMIs.
Your new AMI should look pretty much like this:
AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
Name: Ubuntu Server 12.04.1 LTS
Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
Number of Instances: 1
Availability Zone: No Preference
Instance Type: Micro (t1.micro)
Instance Class: On Demand Edit Instance Details
EBS-Optimized: No
Monitoring: Disabled Termination Protection: Disabled
Tenancy: Default
Kernel ID: Use Default Shutdown Behavior: Stop
RAM Disk ID: Use Default
Network Interfaces:
Secondary IP Addresses:
User Data:
IAM Role: Edit Advanced Details
Key Pair Name: CSG Edit Key Pair
Security Group(s): sg-a098e9c8 Edit Firewall
Test the new AMI
Launch a new AMI instance and check that files are in the correct places. In the EC2 Management Console do:
EC2 DashBoard -> AMIs -> Select CSG instance -> Launch Instance
Launch Instances (take defaults)
Advanced Instance Options (take defaults)
Storage Device Configuration -> Edit
Change volume to 30G or larger -> Continue # Defaults are OK
Instance Details
Key Name = test of instance
Create Key/Pair if you need to, most likely you can use one you have created
Choose a Security Group -> sg-a098e9c8 - quick-start-1
Review -> Launch
