There is a server which is offline and doesn't have any connection to the Internet.

Then we need station (or laptop / virtual machine), which has the same major Red Hat Enterprise Linux version as server and is connected to the Red Hat Network/Proxy/Satellite.

• Copy the /var/lib/rpm to the station connected to the Internet (you can use USB/CD…)

  scp -r /var/lib/rpm root@station:/tmp/
  

• Install the download only plugin for yum and createrepo on the machine which is connected to the Internet (Red Hat Network):

  yum install yum-downloadonly createrepo
  yum clean all
  

• Backup the original rpm directory on the station and replace it with the rpm directory from the "offline" server:

  mv -v /var/lib/rpm /var/lib/rpm.orig
  mv -v /tmp/rpm /var/lib/
  

• Download updates to /tmp/rpm_updates and return back the /var/lib/rpm

  mkdir -v /tmp/rpm_updates
  yum update --downloadonly --downloaddir /tmp/rpm_updates
  createrepo /tmp/rpm_updates
  rm -rvf /var/lib/rpm
  mv -v /var/lib/rpm.orig /var/lib/rpm
  

• Transfer the downloaded rpms to the server and update:

  scp -r /tmp/rpm_updates root@server:/tmp/
  ssh root@server
  
  cat > /etc/yum.repos.d/rhel-offline-updates.repo << \EOF
  [rhel-offline-updates]
  name=Red Hat Enterprise Linux $releasever - $basearch - Offline Updates Repository
  baseurl=file:///tmp/rpm_updates
  enabled=1
  EOF
  
  yum upgrade
  

…and the server is updated.

These updates are the same as if "yum update" had been executed on a station that had a connection to the Internet.

Support for the iLO3 fence device has been added to the fence_ipmilan fence device in the following errata: http://rhn.redhat.com/errata/RHEA-2010-0876.html.

The iLO3 firmware should be a minimum of 1.15 as provided by HP.

On both cluster nodes, install the following OpenIPMI packages used for fencing:

$ yum install OpenIPMI OpenIPMI-tools

Stop and disable the 'acpid' daemon:

$ service acpid stop; chkconfig acpid off

Test ipmitool interaction with iLO3:

$ ipmitool -H <iloip> -I lanplus -U <ilousername> -P <ilopassword> chassis power status

The desired output is:

Chassis Power is on

<?xml version="1.0"?> 

<cluster alias="rh5nodesThree" configversion="32" name="rh5nodesThree"> 

    <fencedaemon cleanstart="0" postfaildelay="1" postjoindelay="3"/> 

    <clusternodes> 

        <clusternode name="rh5node1.examplerh.com" nodeid="1" votes="1"> 

            <fence> 

                <method name="1"> 

                    <device domain="rh5node1" name="ilo3node1"/> 

                </method> 

            </fence> 

        </clusternode> 

        <clusternode name="rh5node2.examplerh.com" nodeid="2" votes="1"> 

            <fence> 

                <method name="1"> 

                    <device domain="rh5node2" name="ilo3node2"/> 

                </method> 

            </fence> 

        </clusternode> 

        <clusternode name="rh5node3.examplerh.com" nodeid="3" votes="1"> 

            <fence> 

                <method name="1"> 

                    <device domain="rh5node3" name="ilo3node3"/> 

                </method> 

            </fence> 

        </clusternode> 

    </clusternodes> 

    <cman expectedvotes="3"> 

        <multicast addr="229.5.1.1"/> 

    </cman> 

    <fencedevices> 

        <fencedevice agent="fenceipmilan" powerwait="10" ipaddr="XX.XX.XX.XX" lanplus="1" login="username" name="ilo3node1" passwd="password"/> 

        <fencedevice agent="fenceipmilan" powerwait="10" ipaddr="XX.XX.XX.XX" lanplus="1" login="username" name="ilo3node2" passwd="password"/> 

        <fencedevice agent="fenceipmilan" powerwait="10" ipaddr="XX.XX.XX.XX" lanplus="1" login="username" name="ilo3node3" passwd="password"/> 

    </fencedevices> 

    <rm> 

        <failoverdomains/> 

        <resources/> 

    </rm> 

</cluster> 

Test that fencing is successful.  From node1 attempt to fence node2 as follows:

$ fencenode node2

For more information on fencing cluster nodes manually then see the following article: How do you manually call fencing agents from the commandline?

Obviously it becomes quite impractical to search through the /sys filesystem for the relevant files when there is a large variety of Fibre Channel-related information of interest. Instead of manual searching, the systool (1) command provides a simple but powerful means of examining and analyzing this information. Detailed below are several commands which demonstrate samples of information which the systool command can be used to examine.

To examine some simple information about the Fibre Channel HBAs in a machine:

# systool -c fchost -v

To look at verbose information regarding the SCSI adapters present on a system:

# systool -c scsihost -v

To see what Fibre Channel devices are connected to the Fibre Channel HBA cards:

# systool -c fcremoteports -v -d

For Fibre Channel transport information:

# systool -c fctransport -v

For information on SCSI disks connected to a system:

# systool -c scsidisk -v

To examine more disk information including which hosts are connected to which disks:

# systool -b scsi -v

Furthermore, by installing the sg3utils package it is possible to use the sgmap command to view more information about the SCSI map. After installing the package, run:

# modprobe sg


sg_map -x

Finally, to obtain driver information, including version numbers and active parameters, the following commands can be used for the lpfc and qla2xxx drivers respectively:

# systool -m lpfc -v


systool -m qla2xxx -v

ATTENTION: The syntax of the systool (1) command differs across versions of Red Hat Enterprise Linux. Therefore the commands above are only valid for Red Hat Enterprise Linux 5.

What is the "Magic" SysRq key?

According to the Linux kernel documentation:

It is a 'magical' key combo you can hit which the kernel will respond to regardless of whatever else it is doing, unless it is completely locked up.

The sysrq key is one of the best (and sometimes the only) way to determine what a machine is really doing. It is useful when a system appears to be "hung" or for diagnosing elusive, transient, kernel-related problems.

How do I enable and disable the SysRq key?

For security reasons, Red Hat Enterprise Linux disables the SysRq key by default. To enable it, run:

# echo 1 > /proc/sys/kernel/sysrq

To disable it:

# echo 0 > /proc/sys/kernel/sysrq

To enable it permanently, set the kernel.sysrq value in /etc/sysctl.conf to 1. That will cause it to be enabled on reboot.

# grep sysrq /etc/sysctl.conf
kernel.sysrq = 1

Since enabling sysrq gives someone with physical console access extra abilities, it is recommended to disable it when not troubleshooting a problem or to ensure that physical console access is properly secured.

How do I trigger a sysrq event?

There are several ways to trigger a sysrq event. On a normal system, with an AT keyboard, sysrq events can be triggered from the console with the following key combination:

Alt+PrintScreen+[CommandKey]

For instance, to tell the kernel to dump memory info (command key "m"), you would hold down the Alt and Print Screen keys, and then hit the m key.

Note that this will not work from an X Window System screen. You should first change to a text virtual terminal. Hit Ctrl+Alt+F1 to switch to the first virtual console prior to hitting the sysrq key combination.

On a serial console, you can achieve the same effect by sending a Breaksignal to the console and then hitting the command key within 5 seconds. This also works for virtual serial console access through an out-of-band service processor or remote console like HP iLO, Sun ILOM and IBM RSA. Refer to service processor specific documentation for details on how to send a Breaksignal; for example, How to trigger SysRq over an HP iLo Virtual Serial Port (VSP).

If you have a root shell on the machine (and the system is responding enough for you to do so), you can also write the command key character to the/proc/sysrq-trigger file. This is useful for triggering this info when you are not on the system console or for triggering it from scripts.

# echo 'm' > /proc/sysrq-trigger

When I trigger a sysrq event that generates output, where does it go?

When a sysrq command is triggered, the kernel will print out the information to the kernel ring buffer and to the system console. This information is normally logged via syslog to /var/log/messages.

Unfortunately, when dealing with machines that are extremely unresponsive, syslogd is often unable to log these events. In these situations, provisioning a serial console is often recommended for collecting the data.

What sort of sysrq events can be triggered?

There are several sysrq events that can be triggered once the sysrq facility is enabled. These vary somewhat between kernel versions, but there are a few that are commonly used:

• m - dump information about memory allocation

• t - dump thread state information

• p - dump current CPU registers and flags

• c - intentionally crash the system (useful for forcing a disk or netdump)

• s - immediately sync all mounted filesystems

• u - immediately remount all filesystems read-only

• b - immediately reboot the machine

• o - immediately power off the machine (if configured and supported)

• f - start the Out Of Memory Killer (OOM)

• w - dumps tasks that are in uninterruptable (blocked) state