Goodbye VMWare. Hello AWS

One of the struggling issues(as currently), is the ability to SSH to EC2 environment using names. DNS name in EC2 environment is too long.  You cannot simply remember EC2 ip addresses. They both change once you reboot or terminate the instance. I feel that one thing we can control is the tag names. Even when the instance is rebooted, the tag names will still be the same.

So, I came up with a small script trying to access the tag name as the host for ssh.


Here is the script.


#!/bin/sh
echo "Enter tag name:"
read tag
instance=`ec2-describe-tags | grep "$tag" | cut -f3`
publicip=`ec2-describe-instances $instance | cut -f17 | tr -d '\n'`
echo $publicip
ssh -i yourpublickey.pem ubuntu@$publicip



Change it to match your environment.

Extending LVM partition on my VMware guest

/dev/mapper/VolGroup00-LogVol00
                      19172036  16489208   1693236  91% /
/dev/sda1               101086     16303     79564  18% /boot
tmpfs                   521396         0    521396   0% /dev/shm
-----------------------------------------------------------------

So I have some space left due to "thin" provision for my VMware guest. I created the physical drive(/dev/sdb3) by fdisk and then added to the existing LVM as follow:


[root@php-bob ~]# pvcreate /dev/sda3
  Physical volume "/dev/sda3" successfully created
[root@php-bob ~]# vgextend VolGroup00 /dev/sda3
  Volume group "VolGroup00" successfully extended
[root@php-bob ~]# lvextend -L+25G /dev/VolGroup00/LogVol00
  Extending logical volume LogVol00 to 43.88 GB
  Logical volume LogVol00 successfully resized
[root@php-bob ~]# resize2fs /dev/VolGroup00/LogVol00
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/VolGroup00/LogVol00 is mounted on /; on-line resizing required
Performing an on-line resize of /dev/VolGroup00/LogVol00 to 11501568 (4k) blocks.
The filesystem on /dev/VolGroup00/LogVol00 is now 11501568 blocks long.

--------------------------------------------

/dev/mapper/VolGroup00-LogVol00
                      44565560  16497340  25768228  40% /
/dev/sda1               101086     16303     79564  18% /boot
tmpfs                   521396         0    521396   0% /dev/shm

P2V Conversion with kernel panic errors.

Assuming that you booted into a rescue CD, did a mkinird(with proper options) command, and still get the kernel panic error. Try this:

Comment out all the existing enties for these hostadapters and add these:

alias scsi_hostadapter mptbase
alias scsi_hostadapter mptspi
alias scsi_hostadapter2 ata_piix

BIOS RAID

Getting this error trying to install a new OS over an old hard drive?


Disk contains BIOS metadata, but is not part of any recognized BIOS RAID sets. Ignoring disk sda


dmraid -r -E /dev/sda

Offline uncorrectable sectors

So my task was to figure out what's wrong with a server on a remote site that is still running but no longer accepting file transfer...hmm.

Device: /dev/sda, 1 Currently unreadable (pending) sectors

Offline uncorrectable sectors

Here is temp fix to get it going while planning to replace the drive.

1. smartctl -l selftest /dev/sda

Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 90% 2703 1113543272

2. fdisk -lu /dev/sda

Device Boot      Start         End      Blocks   Id  System

/dev/sda1   *          63      208844      104391   83  Linux

/dev/sda2          208845  1953520064   976655610   8e  Linux LVM

Of course, it has to be /dev/sda2 since the LBA_of_first_error number is within the block range of sda2

Look for Start block # (208845  in my case)

Use this formular:  ((1113543272-208845)*512)/4096=139166803

3. dd if=/dev/zero of=/dev/mapper/VolGroup00-LogVol00 bs=4096 count=1 seek=139166803

My /dev/sda2 is a LVM drive so I use the mapper volume above. I assume that /dev/sda2 will still work. 

Clonezilla Kernel Panic

Clonezilla is nice and all but once in a while, RAID controller differences can make it difficult to boot up the cloned machine. I get kernel panic and the quick fix is to regenerate initrd.

- Use rescue disk
- chroot /home/partimage
- go to /boot
- mkinitrd /boot/initrd-2.6.18-92.el5.img 2.6.18-53.el5PAE (of course, use your kernel version).
- grub-install --recheck /dev/sda (assuming it's sda).


That's it. Reboot and you should be able to boot to your cloned machine.

Remove oldest recovery point from DPM

We have Exchange, Sharepoint, and MSSQL in house too, you know?

Today, one of our recovery point partition ran out of space. Normally, I would just log in and increase the disk size. Well, no more space to increase today.  I looked and there was no way to delete the oldest recovery point from the GUI...wtf?

DPM Management Shell to the rescue.

$pg=get-protectiongroup -dpmserver server01 #type $pg after this command to see result

$ds=get-datasource -protectiongroup $pg[0] #0 is for the first group from the result of $pg

$rp=get-recoverypoint -datasource $ds #type $rp after this command to see result
remove-recoverypoint $rp[0] #0 is for the oldest recoverypoint from the result of $rp

Not pretty, but it solved my problem.

Cloud Computing

So recently, I decided to give cloud computing a chance to see if I can move at least some of our services to the clouds. I have to admit that I was not really into this cloud computing at the beginning. With our current setup, we really don't need to move our database and web applications to clouds. But then I thought about the benefits of have other services that do not depend on the internal database. Services that can be ran on a small database and aren't too critical for our business. Services like informational web sites and file sharing(for demo or presentations).

To be continuted....

VMware converter p2v from IDE to SCSI

My attempt to convert and old physical machine to an ESX environment was involving more work than I thought. Here is how I got it to boot after a number of kernel panics:

Old physical machine: IDE (hda1, hda2, hda3)

VM (converted): SCSI (sda1, sda2, sda3)

1 - I had to boot into rescue mode and edit /etc/modules.conf and add:  alias  scsi_hostadapter BusLogic

2 - Re-create initrd with: mkinitrd -v -f initrd-xxxx.img  xxxx (you should now what xxx stands for)

3 - Of course, re-install grub with grub-install

4 - Make sure grub.conf match the current hdd config (sda)

5 - Make sure my /etc/fstab converted to sda instead of sda stuff

6 - Modify ESX virtual hard disk to use BusLogic. VMware does not like this but it's the only way that I know the VM will boot.

Enable sendxmpp to alert load average

I thought this might be useful to have in addition to your monitoring system.


sendxmpp: You'll have to learn how to install this yourself :) I only tell you when you got it running. Here are some errors I had when I started:
ERROR: Authorization failed: error - not-authorized
Fix:  

Modifying the Protocol.pm of Net::XMPP (line ~1800). Mine is located at ./site_perl/5.8.8/Net/XMPP/Protocol.pm

Comment these lines out.

if($self->{STREAM}->GetStreamFeature($self->GetStreamID(),"xmpp-sasl"))
{
return $self->AuthSASL(%args);
}

Then use: echo "load is high"|sendxmpp -j 10.10.10.222:5222 -u foobar -p ###### -r home  recipient@domain

Puppet on Centos 5

Didn't think it would be so complicated that I have to blog it to remind myself  :)

Setting up puppet master:

1> wget http://puppetlabs.com/downloads/facter/facter-latest.tgz

2> wget http://puppetlabs.com/downloads/puppet/puppet-2.6.0.tar.gz

tar xzvf facter-latest.tgz  (then: ruby install.rb)

tar xzvf puppet-2.6.0.tar.gz (then: ruby install.rb)

Some spelling errors on code:

Run:  puppetmasterd --genconfig|grep factpath

Should look like this:  factpath = /var/lib/puppet/lib/facter:/var/lib/puppet/facts

Should not look: factpath = /var/lib/puppet/lib/facter/${File::PATH_SEPARATOR}$/var/lib/puppet/facts

If bad factpath, edit default.rb in lib/puppet directory and reinstall. Grep for PATH_SEPARATOR and replace the line with:

:factpath => {:default => "$vardir/lib/facter:$vardir/facts",

Go to the conf dir and copy the server.init to /etc/init.d

Make sure /etc/hosts on both client and server know how to ping each other.

Mount your vmware vmdk file(s).

Why do people make it harder than it should be in their howtos?

Here is how to mount your vmware vmdk files.


1. List your partitions:  
vmware-mount.pl -p Servername.vmdk

2. Mount the partition number you want(2):  
vmware-mount Servername.vmdk 2  /mount/point

ClipBucket installation

Add this to your yum repo.
----------------------------------------------------------------
[dag]
name=Dag RPM Repository for Red Hat Enterprise Linux
baseurl=http://apt.sw.be/redhat/el$releasever/en/$basearch/dag
gpgcheck=0
enabled=1
 ----------------------------------------------------------------
Accept and install all dependencies for these:


#yum install ffmpeg
#yum install ffmpeg-devel
#yum install flvtool2

Then, download this:
wget http://mesh.dl.sourceforge.net/sourceforge/ffmpeg-php/ffmpeg-php-0.5.3.1.tbz2

Extract it and then go to ffmpeg-php to compile and install it.

#cd ffmpeg-php
#phpize (this shold
./configure
#make

*** misspell problemat make:  mv ffmpeg_frame.loT ffmpeg_frame.lo   DO NOT make clean AFTER THIS. Just run 'make' again to finish the compilation step.

#make install

That's it. The hard part now is to import the database. Go here to download:

http://clipbucket.org/scm/viewvc.php/trunk/sql/?root=clipbucket

Make sure you're downloading the correct version number for your MySQL.

Import your database schema and edit the config environment to match your server. These are the important ones:

(3, 'baseurl', 'http://domain.tld'),
(4, 'basedir', '/home/user/public_html'),
(11, 'ffmpegpath', '/usr/bin/ffmpeg'),
(12, 'flvpath', '/usr/bin/flvtool2'),
(19, 'mencoderpath', '/usr/bin/mencoder'),
(22, 'mplayerpath', '/usr/bin/mplayer'),
(25, 'php_path', '/usr/bin/php'),

1001 days uptime.

My email server hits 1001 days of uptime!

Old LVM drive

What to do if you have an old LVM with the same VolGroup?

Using a liveCD. I am using Ubuntu. Must have internet access.
unplug the new drive(s) and leave the old drive that you want to recover files from.
Boot it up with the liveCD.
If you're like me who does not like sudo, do this:
# sudo passwd (to change to whatever password you want for root).
# su - (to get in as root).
# apt-get install lvm2
# vgscan
------------------
Found volume group "VolGroup00" using metadata type lvm2
------------------
# vgrename VolGroup00 VolGroupRecover

Shutdown your machine. Plug everything back. When you're back as root:
# vgscan
------------------
Found volume group "VolGroupRecover" using metadata type lvm2
Found volume group "VolGroup00" using metadata type lvm2
------------------
That's what you want to see. Now reactivate them.
# vgchange -a y
# lvscan
You should see the ACTIVE /dev/VolGroupRecover/LogVol--
Well, mount that one!

Monitoring RAID on Linux

So you setup RAID on Linux and got it to work the way you wanted. Now, you're wondering how would you know if one of your drive went bad. You're thinking that the light on drive tray will blink. Well, are you sure it will blink?

Most of my servers are using Adaptec so here is what I did.

Google: download Adaptec Storage Manager
Pick a page that linked to Adaptec site. I downloaded asm_linux_x86_v5_30_17509.rpm
rpm -ivh asm_linux_x86_v5_30_17509.rpm
Don't run this: /usr/StorMan/StorMan.sh (I am using SSH so that won't work since it's a GUI).
I ran this: /usr/StorMan/arcconf
If it's complaining about libstdc++.so.5
Fix: export LD_PRELOAD=/usr/StorMan/libstdc++.so.5
Run arcconf again and you'll see the howto
I ran the line below for my card:
/usr/StorMan/arcconf GETCONFIG 1

Controllers found: 1
----------------------------------------------------------------------
Controller information
----------------------------------------------------------------------
Controller Status : Optimal
Channel description : SATA
Controller Model : SMC Adaptec 2020SA
Controller Serial Number : 267647
Physical Slot : 3
Installed memory : 64 MB
Copyback : Disabled
Background consistency check : Disabled
Automatic Failover : Enabled
Defunct disk drive count : 0
Logical devices/Failed/Degraded : 4/0/0
--------------------------------------------------------
Controller Version Information
--------------------------------------------------------
.
.
.
... and a whole lot more RAID info.

EXT4

I've been hearing all the hypes about ext4 and how fast it boots up your OS so I am about to try it out. Linux kernel 2.6.28 has it in there.

make menuconfig
make
make modules
make modules_install
make install

Well, that's nice. 'make install' did everything for me in grub. I thought I will have to do it the manual way back in the 2.4.xx days.

Hmm I need the latest e2fsprogs (1.41.3). Downloaded and compiled and there it is, mkfs.ext4.

Trying it on a spare drive:

mkfs.ext4 -j /dev/hdb1
mount -t ext4 /dev/hdb1 /mnt/hdb1/

It mounted w/o any prpblem. I'll see if this is any better than ext3.

Well, that's it for now.

-foo

New discoveries

Virtualization!

Not sure if that's a real word but it seems that everyone is going virtual lately. I've been enjoying VMware for the past few years and lately just found out VirtualBox (http://www.virtualbox.org/).

VirtualBox seems much faster than VMware. There seems to be a minor problem with NAT in XP host environment and Ubuntu as guest. Well, it turned out that it was not their fault. It's me who did not RTFM. I wasn't able to ping out so I thought this must be some routing problems. It wasn't. It was the DNS entry that Ubuntu placed it under the default setting. I changed the DNS entry to use our DNS servers by editing /etc/resolv.conf. That did it.

Sun just bought VirtualBox. Hopefully, they will continue improve this little app. I am enjoying it.

Thinstall is another one(http://www.thinstall.com). This one takes you to another level. Basically, your application will run in a virtual environment of its own, independent from the OS that it's running on top.

Say, you want to run Outlook under Thinstall. You basically create your own executable of Outlook and run it on either Vista, XP, or Win2K without any problem. It doesn't even care about licensing. I bet this will go away soon. VMware is now in control of Thinstall.

-foo

Linux Desktops

I've ran into too many articles about Linux desktop lately. They often named - Is this the year of Linux?, The future of Linux desktop, Linux desktop versus Windows, Dumping Vista for Linux, etc...

I am making a living using Linux. I compile, build, hack Linux almost everyday. Don't get me wrong. I love Linux. It's stable, configurable, easy to admin. I am just tired of hearing about Linux desktops.

To be honest, I don't really care about the future of Linux desktop. Maybe I am selfish. I guess I am in a comfort zone with Linux running on the server end. I don't like Linux loosing focus. That's how I also feel about the huge number of distributions out there. Why? Why wasting your time on that but instead of focus on just a few distributions?

Going to the desktop direction, to me, is the popularity contest. I rather stay out of it. I enjoy the Apple vs Microsoft commercials.

Network monitoring

Today, I ran into a nice app for network monitoring, iftop. It is now a new member in my favorite network traffic monitoring list:


iptraf
airtraf
airsnort
ettercap
ntop
iftop
etherape
ethereal