Offline uncorrectable sectors

So my task was to figure out what's wrong with a server on a remote site that is still running but no longer accepting file transfer...hmm.

Device: /dev/sda, 1 Currently unreadable (pending) sectors

Offline uncorrectable sectors

Here is temp fix to get it going while planning to replace the drive.

1. smartctl -l selftest /dev/sda

Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 90% 2703 1113543272

2. fdisk -lu /dev/sda

Device Boot      Start         End      Blocks   Id  System

/dev/sda1   *          63      208844      104391   83  Linux

/dev/sda2          208845  1953520064   976655610   8e  Linux LVM

Of course, it has to be /dev/sda2 since the LBA_of_first_error number is within the block range of sda2

Look for Start block # (208845  in my case)

Use this formular:  ((1113543272-208845)*512)/4096=139166803

3. dd if=/dev/zero of=/dev/mapper/VolGroup00-LogVol00 bs=4096 count=1 seek=139166803

My /dev/sda2 is a LVM drive so I use the mapper volume above. I assume that /dev/sda2 will still work. 

Clonezilla Kernel Panic

Clonezilla is nice and all but once in a while, RAID controller differences can make it difficult to boot up the cloned machine. I get kernel panic and the quick fix is to regenerate initrd.

- Use rescue disk
- chroot /home/partimage
- go to /boot
- mkinitrd /boot/initrd-2.6.18-92.el5.img 2.6.18-53.el5PAE (of course, use your kernel version).
- grub-install --recheck /dev/sda (assuming it's sda).


That's it. Reboot and you should be able to boot to your cloned machine.

Remove oldest recovery point from DPM

We have Exchange, Sharepoint, and MSSQL in house too, you know?

Today, one of our recovery point partition ran out of space. Normally, I would just log in and increase the disk size. Well, no more space to increase today.  I looked and there was no way to delete the oldest recovery point from the GUI...wtf?

DPM Management Shell to the rescue.

$pg=get-protectiongroup -dpmserver server01 #type $pg after this command to see result

$ds=get-datasource -protectiongroup $pg[0] #0 is for the first group from the result of $pg

$rp=get-recoverypoint -datasource $ds #type $rp after this command to see result
remove-recoverypoint $rp[0] #0 is for the oldest recoverypoint from the result of $rp

Not pretty, but it solved my problem.

Cloud Computing

So recently, I decided to give cloud computing a chance to see if I can move at least some of our services to the clouds. I have to admit that I was not really into this cloud computing at the beginning. With our current setup, we really don't need to move our database and web applications to clouds. But then I thought about the benefits of have other services that do not depend on the internal database. Services that can be ran on a small database and aren't too critical for our business. Services like informational web sites and file sharing(for demo or presentations).

To be continuted....

VMware converter p2v from IDE to SCSI

My attempt to convert and old physical machine to an ESX environment was involving more work than I thought. Here is how I got it to boot after a number of kernel panics:

Old physical machine: IDE (hda1, hda2, hda3)

VM (converted): SCSI (sda1, sda2, sda3)

1 - I had to boot into rescue mode and edit /etc/modules.conf and add:  alias  scsi_hostadapter BusLogic

2 - Re-create initrd with: mkinitrd -v -f initrd-xxxx.img  xxxx (you should now what xxx stands for)

3 - Of course, re-install grub with grub-install

4 - Make sure grub.conf match the current hdd config (sda)

5 - Make sure my /etc/fstab converted to sda instead of sda stuff

6 - Modify ESX virtual hard disk to use BusLogic. VMware does not like this but it's the only way that I know the VM will boot.

Enable sendxmpp to alert load average

I thought this might be useful to have in addition to your monitoring system.


sendxmpp: You'll have to learn how to install this yourself :) I only tell you when you got it running. Here are some errors I had when I started:
ERROR: Authorization failed: error - not-authorized
Fix:  

Modifying the Protocol.pm of Net::XMPP (line ~1800). Mine is located at ./site_perl/5.8.8/Net/XMPP/Protocol.pm

Comment these lines out.

if($self->{STREAM}->GetStreamFeature($self->GetStreamID(),"xmpp-sasl"))
{
return $self->AuthSASL(%args);
}

Then use: echo "load is high"|sendxmpp -j 10.10.10.222:5222 -u foobar -p ###### -r home  recipient@domain

Puppet on Centos 5

Didn't think it would be so complicated that I have to blog it to remind myself  :)

Setting up puppet master:

1> wget http://puppetlabs.com/downloads/facter/facter-latest.tgz

2> wget http://puppetlabs.com/downloads/puppet/puppet-2.6.0.tar.gz

tar xzvf facter-latest.tgz  (then: ruby install.rb)

tar xzvf puppet-2.6.0.tar.gz (then: ruby install.rb)

Some spelling errors on code:

Run:  puppetmasterd --genconfig|grep factpath

Should look like this:  factpath = /var/lib/puppet/lib/facter:/var/lib/puppet/facts

Should not look: factpath = /var/lib/puppet/lib/facter/${File::PATH_SEPARATOR}$/var/lib/puppet/facts

If bad factpath, edit default.rb in lib/puppet directory and reinstall. Grep for PATH_SEPARATOR and replace the line with:

:factpath => {:default => "$vardir/lib/facter:$vardir/facts",

Go to the conf dir and copy the server.init to /etc/init.d

Make sure /etc/hosts on both client and server know how to ping each other.