Creating an AMI on EC2 - Revision history

Mktrost: /* Cleanup Instance for AMI Creation */

2014-10-14T18:54:08Z

Cleanup Instance for AMI Creation

← Older revision		Revision as of 14:54, 14 October 2014
Line 103:		Line 103:

	==Cleanup Instance for AMI Creation==		==Cleanup Instance for AMI Creation==
			First time from generic/starcluster AMI
	# Disable password-based logins for root		# Disable password-based logins for root
	## Open /etc/ssh/sshd_config		## Open /etc/ssh/sshd_config
Line 110:		Line 111:


			Each time we generate a new AMI, run:
			<pre>sudo shred -u /etc/ssh/_key /etc/ssh/_key.pub
			sudo find / -name "authorized_keys" -exec rm -f {} \;
			rm -rf ~/.ssh
			shred -u ~/.*history
			sudo find /root/.history /home//.*history -exec rm -f {} \;
			history -w
			history -c
			</pre>
			These commands do the following:
	# Remove SSH host key pairs		# Remove SSH host key pairs
	~~## <code>sudo shred -u /etc/ssh/_key /etc/ssh/_key.pub</code>~~
	# Remove SSH authorized keys		# Remove SSH authorized keys
	~~##<code>sudo find / -name "authorized_keys" -exec rm -f {} \;</code>~~
	# Remove ssh		# Remove ssh
	~~##<code>rm -rf ~/.ssh</code>~~
	# Delete shell history		# Delete shell history
	~~<pre>shred -u ~/.*history~~
	~~sudo find /root/.history /home//.*history -exec rm -f {} \;~~
	~~history -w~~
	~~history -c~~
	~~</pre>~~

	== Create the AMI ==		== Create the AMI ==

Mktrost: /* Cleanup Instance for AMI Creation */

2014-10-14T18:30:51Z

Cleanup Instance for AMI Creation

← Older revision		Revision as of 14:30, 14 October 2014
Line 117:		Line 117:
	##<code>rm -rf ~/.ssh</code>		##<code>rm -rf ~/.ssh</code>
	# Delete shell history		# Delete shell history
	## <~~code~~> shred -u ~/.*history~~</code>~~		<pre>shred -u ~/.*history
	~~##<code>~~sudo find /root/.history /home//.*history -exec rm -f {} \;</~~code~~>		sudo find /root/.history /home//.*history -exec rm -f {} \;
			history -w
			history -c
			</pre>

	== Create the AMI ==		== Create the AMI ==

Mktrost: /* Cleanup Instance for AMI Creation */

2014-10-14T18:26:24Z

Cleanup Instance for AMI Creation

← Older revision		Revision as of 14:26, 14 October 2014
Line 108:		Line 108:
	# Disable root access		# Disable root access
	## <code> sudo passwd -l root</code>		## <code> sudo passwd -l root</code>


	# Remove SSH host key pairs		# Remove SSH host key pairs
	## <code>sudo shred -u /etc/ssh/_key /etc/ssh/_key.pub</code>		## <code>sudo shred -u /etc/ssh/_key /etc/ssh/_key.pub</code>
	# Remove SSH authorized keys		# Remove SSH authorized keys
	##<code>sudo find / -name "authorized_keys" -exec rm -f {} \;</code>		##<code>sudo find / -name "authorized_keys" -exec rm -f {} \;</code>
			# Remove ssh
			##<code>rm -rf ~/.ssh</code>
	# Delete shell history		# Delete shell history
	## <code> shred -u ~/.*history</code>		## <code> shred -u ~/.*history</code>

Mktrost: /* Cleanup Instance for AMI Creation */

2014-10-14T18:21:29Z

Cleanup Instance for AMI Creation

← Older revision		Revision as of 14:21, 14 October 2014
Line 109:		Line 109:
	## <code> sudo passwd -l root</code>		## <code> sudo passwd -l root</code>
	# Remove SSH host key pairs		# Remove SSH host key pairs
	~~## Remove host key pairs located in <code>/etc/ssh</code> (ssh_host_dsa_key, ssh_host_dsa_key.pub, ssh_host_key, ssh_host_key.pub, ssh_host_rsa_key, ssh_host_rsa_key.pub)~~
	## <code>sudo shred -u /etc/ssh/_key /etc/ssh/_key.pub</code>		## <code>sudo shred -u /etc/ssh/_key /etc/ssh/_key.pub</code>
	# Remove SSH authorized keys		# Remove SSH authorized keys

Mktrost at 14:26, 14 October 2014

2014-10-14T14:26:13Z

@@ Line 81: / Line 81: @@
    swapon -s       # Should show the swap device
 </code>
 === Create the AMI ===
@@ Line 97: / Line 100: @@
 # <code>cd</code>
 # Go to : [[#Create the AMI|Create the AMI]]
 == Create the AMI ==

Mktrost at 20:35, 10 October 2014

2014-10-10T20:35:54Z

← Older revision		Revision as of 16:35, 10 October 2014
Line 114:		Line 114:


			== Older/Additional Instructions ==
	== Install the Software ==		=== Install the Software ===

	'''(1)''' There are a number of additional Debian packages that you may well need, so we make		'''(1)''' There are a number of additional Debian packages that you may well need, so we make
Line 194:		Line 194:
	Run the tests to be sure everything is OK.		Run the tests to be sure everything is OK.

	== Configure the Host to be Usable ==		=== Configure the Host to be Usable ===

	It is useful to configure /etc/rc.local to do most things you need at boot time.		It is useful to configure /etc/rc.local to do most things you need at boot time.
Line 278:		Line 278:
	</code>		</code>

	== Test the new AMI ==		=== Test the new AMI ===

	Launch a new AMI instance and check that files are in the correct places.		Launch a new AMI instance and check that files are in the correct places.

Mktrost at 20:34, 10 October 2014

2014-10-10T20:34:39Z

← Older revision		Revision as of 16:34, 10 October 2014
Line 7:		Line 7:
	These notes assume you have already created an EC2 account and have the certificates and keys set up properly.		These notes assume you have already created an EC2 account and have the certificates and keys set up properly.

	== Launch an instance ==
			== Create new GotCloud AMI from StarCluster AMI ==
			=== Launch an instance ===

	<code>		<code>
Line 34:		Line 36:
	# Select the key/pair you want to use & Launch		# Select the key/pair you want to use & Launch

	== Setup the instance with GotCloud ==		=== Setup the instance with GotCloud ===
	This assumes you have already logged onto the instance.		This assumes you have already logged onto the instance.

Line 60:		Line 62:
	#* <code>:q</code>		#* <code>:q</code>

	== ~~Create the AMI~~ ==		=== Set Up Swap Space ===

	~~Once your instance~~ is ~~all ready with everything~~ you ~~want, create the AMI.~~		Issue the command '''swapon -s''' to see if there is swap space.
			If there is only a header line, you need to add a swap file like this:

	~~In your browser at the EC2 Management Console do the following:~~		<code>
	# ~~Select the running instance~~		df -h # Be sure there's enough space, decide on swap size
	# ~~Right click, <code>~~Create ~~Image<~~/~~code>~~		# Create a file /swap to use (assuming / is large enough)
	# ~~Enter name & Description~~		sudo bash # Run these commands as root
	~~# Ensure volume size is correct~~		swap=/swap
	# ~~Mark delete~~ on ~~terminate~~		dd if=/dev/zero of=$swap bs=524288 count=16384 # 8GB swap on t1.micro 15G=bs=1073741824 count=15
	#:~~This will take several minutes to complete.~~		chown root:root $swap
	~~#:In the EC2 Dashboard, you can monitor the progress.~~		mkswap $swap
	#~~:When it is done, you'll see a new AMI under~~ the ~~list of AMIs.~~		chmod 0600 $swap
	~~# When completed, terminate your old instance~~		swapon $swap
			echo "$swap none swap sw 0 0" >> /etc/fstab

			swapon -s # Should show the swap device
			</code>

	~~Your new~~ AMI ~~should look pretty much like this~~:		=== Create the AMI ===
			# Go to : [[#Create the AMI\|Create the AMI]]


Line 88:		Line 96:
	#* Specify <code>-j #</code> based on the number of CPUs your instance has		#* Specify <code>-j #</code> based on the number of CPUs your instance has
	# <code>cd</code>		# <code>cd</code>
			# Go to : [[#Create the AMI\|Create the AMI]]

			== Create the AMI ==

	~~== Set Up Swap Space ==~~		Once your instance is all ready with everything you want, create the AMI.

			In your browser at the EC2 Management Console do the following:
			# Select the running instance
			# Right click, <code>Create Image</code>
			# Enter name & Description
			# Ensure volume size is correct
			# Mark delete on terminate
			#:This will take several minutes to complete.
			#:In the EC2 Dashboard, you can monitor the progress.
			#:When it is done, you'll see a new AMI under the list of AMIs.
			# When completed, terminate your old instance

	~~Issue the command '''swapon -s''' to see if there is swap space.~~
	~~If there is only a header line, you need to add a swap file like this:~~

	~~<code>~~
	~~df -h # Be sure there's enough space, decide on swap size~~
	~~# Create a file /swap to use (assuming / is large enough)~~
	~~sudo bash # Run these commands as root~~
	~~swap=/swap~~
	~~dd if=/dev/zero of=$swap bs=524288 count=16384 # 8GB swap on t1.micro 15G=bs=1073741824 count=15~~
	~~chown root:root $swap~~
	~~mkswap $swap~~
	~~chmod 0600 $swap~~
	~~swapon $swap~~
	~~echo "$swap none swap sw 0 0" >> /etc/fstab~~

	~~swapon -s # Should show the swap device~~
	~~</code>~~

	== Install the Software ==		== Install the Software ==

Mktrost at 17:29, 10 October 2014

2014-10-10T17:29:59Z

← Older revision		Revision as of 13:29, 10 October 2014
Line 21:		Line 21:
	## Enter in the search box: <code>starcluster-base-ubuntu</code>		## Enter in the search box: <code>starcluster-base-ubuntu</code>
	## Select: <code>starcluster-base-ubuntu-12.04-x86_64 - ami-765b3e1f</code>		## Select: <code>starcluster-base-ubuntu-12.04-x86_64 - ami-765b3e1f</code>
	# Select the Instance Type: <code>~~Micro t1~~.~~micro~~</code>		# Select the Instance Type: <code>Compute optimized c3.2xlarge</code>
	#* ~~Since we are just using this instance to setup the system~~, ~~pick the smallest/cheapest~~ machine.		#* You can use a smaller/cheaper machine - I originaly used t1.micro, but I found things go so much faster with a larger machine.
	# Click: <code>Review and Launch</code>		# Click: <code>Review and Launch</code>
	## Select: <code>Make General Purpose (SSD) the boot volume for this instance.</code>		## Select: <code>Make General Purpose (SSD) the boot volume for this instance.</code>
Line 39:		Line 39:
	# Get the latest version of GotCloud:		# Get the latest version of GotCloud:
	#* Multiples ways to do this, one way is to do:		#* Multiples ways to do this, one way is to do:
	#*# <code>git clone https://github.com/statgen/gotcloud.git ~~gc_src~~</code>		#*# <code>sudo git clone https://github.com/statgen/gotcloud.git</code>
	# Download cmake (required to build premo)		# Download cmake (required to build premo)
	#*<code>sudo apt-get update</code>		#*<code>sudo apt-get update</code>
Line 45:		Line 45:
	#*<code>sudo apt-get install cmake</code>		#*<code>sudo apt-get install cmake</code>
	## Build the source (if you obtained the source code).		## Build the source (if you obtained the source code).
	### <code>cd ~~gc_src~~/src</code>		### <code>cd gotcloud/src</code>
	### <code>make</code>		### <code>sudo make</code>
	### <code>~~cd ..~~</code>		###* Specify <code>-j #</code> based on the number of CPUs your instance has, if more than 1
	~~## Generate~~ the ~~installation.~~
	~~### <code>cd gc_src/</code>~~
	~~### <code>./debian/makedeb.sh bin =</code>~~
	### <code>cd</code>		### <code>cd</code>
	~~# Install: <code>sudo dpkg -i gc_src/gotcloud-bin_1.14.3_amd64.deb</code>~~
	~~# Move installation to home so will be on all nodes: <code>sudo mv /usr/local/gotcloud .</code>~~
	~~# Remove source: <code>rm -rf gc_src/</code> (required since not enough storage space.)~~
	# Get the reference files		# Get the reference files
	## wget ftp://anonymous@share.sph.umich.edu/gotcloud/ref/h37-db135-v3.tgz		## wget ftp://anonymous@share.sph.umich.edu/gotcloud/ref/h37-db135-v3.tgz
Line 60:		Line 54:
	# Move reference to gotcloud directory: <code>sudo mv gotcloud.ref gotcloud</code>		# Move reference to gotcloud directory: <code>sudo mv gotcloud.ref gotcloud</code>
	# Remove tar file: <code>rm h37-db135-v3.tgz</code>		# Remove tar file: <code>rm h37-db135-v3.tgz</code>
	# ~~Add aliases for GotCloud and tabix~~: vi .~~bash_aliases~~		# Set the paths, by updating .profile: <code>vi .profile</code>
	## <~~code~~>~~alias~~ gotcloud='/~~home~~/~~ubuntu~~/gotcloud/bin/gotcloud'</code>		#* <code>i</code>
	## <code>~~alias tabix='/home/ubuntu/gotcloud/bin/tabix'~~</code>		#: <pre>if [ -d "$HOME/gotcloud" ] ; then PATH="$HOME/gotcloud:$PATH" fi if [ -d "$HOME/gotcloud/bin" ] ; then PATH="$HOME/gotcloud/bin:$PATH" fi if [ -d "$HOME/gotcloud/scripts" ] ; then PATH="$HOME/gotcloud/scripts:$PATH" fi</pre>
			#* <code>ESC</code>
			#* <code>:q</code>


	~~== Create Image ==~~
	== Create the AMI ==		== Create the AMI ==

Line 83:		Line 77:
	Your new AMI should look pretty much like this:		Your new AMI should look pretty much like this:

	~~<code>~~
	AMI~~: Ubuntu Cloud Guest~~ AMI ~~ID ami-3d4ff254 (x86_64)~~		== Update the GotCloud AMI ==
	~~Name: Ubuntu Server 12~~.~~04.1 LTS~~		# Start an instance of the current GotCloud AMI
	~~Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http:~~//~~www.ubuntu.com~~/~~cloud~~/~~services).~~		#* Suggest an instance with some CPU so you can parallelize the "make" call.
	~~Number of Instances: 1~~		# Login as ubuntu
	~~Availability Zone: No Preference~~		# <code>cd gotcloud</code>
	~~Instance Type: Micro (t1.micro)~~		# <code>sudo git pull</code>
	~~Instance Class: On Demand Edit Instance Details~~		# <code>cd gotcloud/src</code>
	~~EBS~~-~~Optimized: No~~		# <code>sudo make</code>
	~~Monitoring: Disabled Termination Protection: Disabled~~		#* Specify <code>-j #</code> based on the number of CPUs your instance has
	~~Tenancy: Default~~		# <code>cd</code>
	~~Kernel ID: Use Default Shutdown Behavior: Stop~~
	~~RAM Disk ID: Use Default~~
	~~Network Interfaces:~~
	~~Secondary IP Addresses:~~
	~~User Data:~~
	~~IAM Role: Edit Advanced Details~~
	~~Key Pair Name: CSG Edit Key Pair~~
	~~Security Group(s): sg-a098e9c8 Edit Firewall~~
	</code>

	== Set Up Swap Space ==		== Set Up Swap Space ==

Mktrost: /* Create Image */

2014-10-10T00:50:19Z

Create Image

← Older revision		Revision as of 20:50, 9 October 2014
Line 66:		Line 66:

	== Create Image ==		== Create Image ==
			== Create the AMI ==

			Once your instance is all ready with everything you want, create the AMI.

			In your browser at the EC2 Management Console do the following:
			# Select the running instance
			# Right click, <code>Create Image</code>
			# Enter name & Description
			# Ensure volume size is correct
			# Mark delete on terminate
			#:This will take several minutes to complete.
			#:In the EC2 Dashboard, you can monitor the progress.
			#:When it is done, you'll see a new AMI under the list of AMIs.
			# When completed, terminate your old instance

			Your new AMI should look pretty much like this:

			<code>
			AMI: Ubuntu Cloud Guest AMI ID ami-3d4ff254 (x86_64)
			Name: Ubuntu Server 12.04.1 LTS
			Description: Ubuntu Server 12.04.1 LTS with support available from Canonical (http://www.ubuntu.com/cloud/services).
			Number of Instances: 1
			Availability Zone: No Preference
			Instance Type: Micro (t1.micro)
			Instance Class: On Demand Edit Instance Details
			EBS-Optimized: No
			Monitoring: Disabled Termination Protection: Disabled
			Tenancy: Default
			Kernel ID: Use Default Shutdown Behavior: Stop
			RAM Disk ID: Use Default
			Network Interfaces:
			Secondary IP Addresses:
			User Data:
			IAM Role: Edit Advanced Details
			Key Pair Name: CSG Edit Key Pair
			Security Group(s): sg-a098e9c8 Edit Firewall
			</code>

	== Set Up Swap Space ==		== Set Up Swap Space ==

Mktrost: /* Create the AMI */

2014-10-10T00:06:05Z

Create the AMI

@@ Line 248: / Line 248: @@
   fi
   exit 0
 </code>