I recently had a problem on a CentOS-6 machine where my dhcp server did not start at boot because it was serving a virtual network interface which did not yet exist.
The best solution to this problem would be for the dhcp server to start up and wait for the network interface to come up. Many other network tools do this successfully so I don't know why dhcpd should be different. That problem is however too big for me to fix on my server so I need a more simple approach.
To solve this, I would like to adjust my dhcp server to start after the virtual network interface service. CentOS-6 uses the (old) RedHat style init scripts (with some LSB configuration too). RedHat like to proclaim that they use upstarts now but all the upstarts do is make a call to the /etc/rc.d/rc script, just like init used to do.
So, the problem is still based around the System V style scripts which have magic comments which define when to start & stop the scripts.
This is the relevant parts of the header for the dhcp server (/etc/rc.d/init.d/dhcpd):
### BEGIN INIT INFO
# Provides: dhcpd
# Default-Start:
# Default-Stop:
# Should-Start: portreserve
# Required-Start: $network
# Required-Stop:
# Short-Description: Start and stop the DHCP server
# Description: dhcpd provides the Dynamic Host Configuration Protocol (DHCP)
# server.
### END INIT INFO
#
# The fields below are left around for legacy tools (will remove later).
#
# chkconfig: - 65 35
Despite the comments about being legacy, when installed using chkconfig, the priorities used are indeed Start 65, Kill 35. We can confirm this with a simple ls
ls /etc/rc.d/rc?.d/*dhcpd
The virtual network is also controlled by an init script with a more modest header (/etc/rc.d/init.d/vand):
# chkconfig: 2345 95 05
# description: Virtual Area Network Deamon
Not surprisingly, this will Start at 95 and Kill at 05.
The first idea I had was to modify the dhcpd init scrip directly. Unfortunately this script is not marked as config file (in the rpm) so when a new version comes out, my changes will be lost. I need something better than that.
It is not immediately obvious but chkconfig does have the ability to alter the priorities. This is called override (but not to be confused with the command line parameter --override).
I was unable to find and documentation on the override but I did work out that his minimal config file was all that was needed:
/etc/chkconfig.d/dhcp:
### BEGIN INIT INFO
# Required-Start: vand
### END INIT INFO
When chkconfig reads the headers from the dhpcd init script, it will also read this file (because it has the same basename) and override the values from the init script.
All that is needed is to apply the settings with:
chkconfig dhcpd on
Some information about LSB init scripts can be found here http://refspecs.linuxbase.org/LSB_3.1.1/LSB-Core-generic/LSB-Core-generic/initscrcomconv.html
The final job is to make puppet aware of my changes. Perhaps in my next post.
Wednesday, May 29, 2013
Friday, May 24, 2013
What happened to java -client?
On our shared server, users are limited to 25 processes. A login script further reduces the soft limit by another 5 processes. This safety margin allows the user to log in an kill something which is using up all their processes. If they want they can raise their soft limit back up to the hard limit.
This works well for most things but java insists on using lots of processes. Confusingly, it prints this message:
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# An error report file with more information is saved as:
# ./hs_err_pid27297.log
The log file is not of much use. It repeats the claim about 'insufficient memory' which is not true and none of the listed solutions will help.
The problem is that the default garbage collector (GC) scales up with the number of CPU cores. Our old server had 8 but the new server has 24. This is easily pushing the users over the 20 thread soft limit.
I don't if the log file lists the number of java threads. If it does I can't work out how to read it. If it could be found it could be compared with the number of available processes:
expr $(ulimit -u) - $(ps x | wc --lines)
In the past I used to solve this problem by adding this to the login script:
alias java="java -client"
but that does not work any more. I don't know when it was removed but it leaves me with the headache of finding a new workaround.
After some research I found this page https://wiki.csiro.au/pages/viewpage.action?pageId=545034311 which suggested using
-XX:ParallelGCThreads=1 -XX:+UseParallelGC
or even just
-XX:+UseSerialGC
This seems to solve the immediate problem but still is not all that useful. To specify the flag to run java you need this syntax:
java -XX:+UseSerialGC
but for javac you have to specify like this:
javac -J-XX:+UseSerialGC
and I don't expect anyone should have to remember that. One simple solution is to alias both those commands but there are many other tools which are part of java which would also need aliases. These other tools may (and do) use their own unique command line parameters which I would rather not have to learn.
Luckily there is also an environment variable you can set like this:
export _JAVA_OPTIONS="-XX:ParallelGCThreads=1 -XX:+UseParallelGC"
however, that makes all the java programs print out this annoying message:
Picked up _JAVA_OPTIONS: -XX:ParallelGCThreads=1 -XX:+UseParallelGC
And the word on the internet is that there is no way to suppress that message.
My solution to use an alias for the most common tools java and javac while keeping the environment variable for all other programs. At first I tried to use this in my alias:
_JAVA_OPTIONS= /usr/bin/java -XX:+UseSerialGC
but an empty variable still shows the message (and best I can tell, bash won't unset a variable like that). I needed to unset it so I had to combine the env command like this:
env -u _JAVA_OPTIONS -XX:+UseSerialGC
My final profile script looks like this:
export _JAVA_OPTIONS="-XX:ParallelGCThreads=1 -XX:+UseParallelGC"
#export _JAVA_OPTIONS="-XX:+UseSerialGC"
alias java="env -u _JAVA_OPTIONS $(which --skip-alias java) $_JAVA_OPTIONS"
alias javac="env -u _JAVA_OPTIONS $(which --skip-alias javac)$(for i in $_JAVA_OPTIONS ; do echo -n " -J$i" ; done)"
You can select which GC you want. I have demonstrated with the more complex example which has two options to be prefixed for javac.
The java users can now happily compile away without java needing the entire machine resources for every invocation.
This works well for most things but java insists on using lots of processes. Confusingly, it prints this message:
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# An error report file with more information is saved as:
# ./hs_err_pid27297.log
The log file is not of much use. It repeats the claim about 'insufficient memory' which is not true and none of the listed solutions will help.
The problem is that the default garbage collector (GC) scales up with the number of CPU cores. Our old server had 8 but the new server has 24. This is easily pushing the users over the 20 thread soft limit.
I don't if the log file lists the number of java threads. If it does I can't work out how to read it. If it could be found it could be compared with the number of available processes:
expr $(ulimit -u) - $(ps x | wc --lines)
In the past I used to solve this problem by adding this to the login script:
alias java="java -client"
but that does not work any more. I don't know when it was removed but it leaves me with the headache of finding a new workaround.
After some research I found this page https://wiki.csiro.au/pages/viewpage.action?pageId=545034311 which suggested using
-XX:ParallelGCThreads=1 -XX:+UseParallelGC
or even just
-XX:+UseSerialGC
This seems to solve the immediate problem but still is not all that useful. To specify the flag to run java you need this syntax:
java -XX:+UseSerialGC
but for javac you have to specify like this:
javac -J-XX:+UseSerialGC
and I don't expect anyone should have to remember that. One simple solution is to alias both those commands but there are many other tools which are part of java which would also need aliases. These other tools may (and do) use their own unique command line parameters which I would rather not have to learn.
Luckily there is also an environment variable you can set like this:
export _JAVA_OPTIONS="-XX:ParallelGCThreads=1 -XX:+UseParallelGC"
however, that makes all the java programs print out this annoying message:
Picked up _JAVA_OPTIONS: -XX:ParallelGCThreads=1 -XX:+UseParallelGC
And the word on the internet is that there is no way to suppress that message.
My solution to use an alias for the most common tools java and javac while keeping the environment variable for all other programs. At first I tried to use this in my alias:
_JAVA_OPTIONS= /usr/bin/java -XX:+UseSerialGC
but an empty variable still shows the message (and best I can tell, bash won't unset a variable like that). I needed to unset it so I had to combine the env command like this:
env -u _JAVA_OPTIONS -XX:+UseSerialGC
My final profile script looks like this:
export _JAVA_OPTIONS="-XX:ParallelGCThreads=1 -XX:+UseParallelGC"
#export _JAVA_OPTIONS="-XX:+UseSerialGC"
alias java="env -u _JAVA_OPTIONS $(which --skip-alias java) $_JAVA_OPTIONS"
alias javac="env -u _JAVA_OPTIONS $(which --skip-alias javac)$(for i in $_JAVA_OPTIONS ; do echo -n " -J$i" ; done)"
You can select which GC you want. I have demonstrated with the more complex example which has two options to be prefixed for javac.
The java users can now happily compile away without java needing the entire machine resources for every invocation.
Sunday, May 12, 2013
Can a KVM guest find out who it's host is?
Our puppet configuration performs a number of checks on our asset database to make sure things are recorded correctly.
One of the properties we record for virtual machines is their location (which host they are running on).
By default, KVM does not expose this information even though there are many ways it could technically be done.
Using bits and pieces from around the internet I have come up with a process where I can pass the serial number of the host into the guest.
qemu permits you to specify many DMI values on the command line like this:
qemu-kvm ... -smbios type=1,serial="MY-SERIAL"
The virtual machine will see this value and puppet will automatically create a fact with this value.
Unfortunately, libvirt does not use this mechanism and the serial number is blank in the virtual machine. The libvirt specification permits setting a value in the .xml config file but it is still not used.
I would like to insert a value which is the same as the host serial but with a prefix to indicate that this is indeed a virtual machine and then a suffix to make sure the serial is unique (such as the vm name).
Initially I used dmidecode to get the host serial number
dmidecode -s system-serial-number
but to run that you must be root and using sudo from the qemu user turned out to be a PITA. In the end I settled for
/usr/bin/hal-get-property --udi /org/freedesktop/Hal/devices/computer --key system.hardware.serial
which uses dbus but means I can run it without being root.
The next step was a shim around qemu-kvm which could add the command line parameters. On RHEL/CentOS 6, the binary lives in /usr/libexec so I put my wrapper in /usr/local/libexec (I think that is the first time I have ever used that directory). When a vm is being started, the first parameter is -name and then the machine name is specified (at least in the current EL6 version. This was not the case in a previous release so it could change). I check to see if that has been specified because qemu-kvm is also invoked by libvirt to check it's configuration/capabilities which does not require the dmi serial (although it does not hurt).
/usr/local/libexec/qemu-kvm
#!/bin/bash
# This is a wrapper around qemu which will supply
# DMI information
if [ "$1" = "-name" ] ; then
SERIAL=$(/usr/bin/hal-get-property --udi /org/freedesktop/Hal/devices/computer --key system.hardware.serial)
exec /usr/libexec/qemu-kvm "$@" -smbios type=1,serial="KVM-$SERIAL-$2"
else
exec /usr/libexec/qemu-kvm "$@"
fi
The final step is to tell libvirt to use my new shim rather than the qemu-kvm binary directly. This also does not seem optimal but can be done by editing every guest and setting the <emulator> path to /usr/local/libexec/qemu-kvm
(either using virsh edit or your favourite XML editor).
One of the properties we record for virtual machines is their location (which host they are running on).
By default, KVM does not expose this information even though there are many ways it could technically be done.
Using bits and pieces from around the internet I have come up with a process where I can pass the serial number of the host into the guest.
qemu permits you to specify many DMI values on the command line like this:
qemu-kvm ... -smbios type=1,serial="MY-SERIAL"
The virtual machine will see this value and puppet will automatically create a fact with this value.
Unfortunately, libvirt does not use this mechanism and the serial number is blank in the virtual machine. The libvirt specification permits setting a value in the .xml config file but it is still not used.
I would like to insert a value which is the same as the host serial but with a prefix to indicate that this is indeed a virtual machine and then a suffix to make sure the serial is unique (such as the vm name).
Initially I used dmidecode to get the host serial number
dmidecode -s system-serial-number
but to run that you must be root and using sudo from the qemu user turned out to be a PITA. In the end I settled for
/usr/bin/hal-get-property --udi /org/freedesktop/Hal/devices/computer --key system.hardware.serial
which uses dbus but means I can run it without being root.
The next step was a shim around qemu-kvm which could add the command line parameters. On RHEL/CentOS 6, the binary lives in /usr/libexec so I put my wrapper in /usr/local/libexec (I think that is the first time I have ever used that directory). When a vm is being started, the first parameter is -name and then the machine name is specified (at least in the current EL6 version. This was not the case in a previous release so it could change). I check to see if that has been specified because qemu-kvm is also invoked by libvirt to check it's configuration/capabilities which does not require the dmi serial (although it does not hurt).
/usr/local/libexec/qemu-kvm
#!/bin/bash
# This is a wrapper around qemu which will supply
# DMI information
if [ "$1" = "-name" ] ; then
SERIAL=$(/usr/bin/hal-get-property --udi /org/freedesktop/Hal/devices/computer --key system.hardware.serial)
exec /usr/libexec/qemu-kvm "$@" -smbios type=1,serial="KVM-$SERIAL-$2"
else
exec /usr/libexec/qemu-kvm "$@"
fi
The final step is to tell libvirt to use my new shim rather than the qemu-kvm binary directly. This also does not seem optimal but can be done by editing every guest and setting the <emulator> path to /usr/local/libexec/qemu-kvm
(either using virsh edit or your favourite XML editor).
Wednesday, May 1, 2013
ProLiant Virtual Serial Port
While working through the configuration of iLO3 for our HP DL380 G7 servers, I found a few pages talking about the Virtual Serial Port (VSP).
This sounded interesting so I though I would try it out. I was using this page as a guide http://www.fatmin.com/2011/06/redirect-linux-console-to-hp-ilo-via-ssh.html but it was a bit out of date.
The first thing to mention was that this does indeed work with iLO3. When configuring your iLO3, remember that in order to ssh to iLO3 you must have a PTY (-t -t), use protocol version 2 (-2) and use DSA keys (ssh-keygen -t dsa). My old configuration for older versions of iLO used exactly the opposite for all of these settings which took some time to figure out (and HP only recently identified/fixed the bugs with ssh keys).
Once you can ssh to your iLO interface, you can issue the command
VSP
and it will start a terminal emulator (actually, it just passes everything through to your terminal program so you can use xterm if you want but safer to assume a vt102). To exit, press ESCape and then ( and you will return to the iLO prompt.
For the OS configuration, I found that I had to make a few alterations. First, RHEL6/CentOS6 uses upstarts and not inittab. If you are not booting with a serial console you must create your own init script:
/etc/init/ttyS1.conf
# ttyS1 - agetty
#
# This service maintains a agetty on ttyS1 for iLO3 VSP.
stop on runlevel [S016]
start on runlevel [235]
respawn
exec agetty -8 -L -w /dev/ttyS1 115200 vt102
You can then enable the console with the command
start ttyS1
Next time you reboot it should start automatically.
In order to log in as root you must add the terminal it to the /etc/securetty file:
echo '/dev/ttyS1' >> /etc/securetty
So that was all good. Now I want to roll this out to all my servers. To do this I needed some puppet magic. Not surprising, puppet let me down on most of the steps for this.
# Install the init script. This one is easy, just dump in the file
file { "/etc/init/ttyS1.conf":
...
}
# Add the entry to securetty
augeas { "securetty_ttyS1":
# Ha ha. securetty has a special lens which is different to most other config files
context => "/files/etc/securetty",
changes => [ "ins 0 before /files/etc/securetty/1",
"set /files/etc/securetty/0 ttyS1",
],
onlyif => "match *[.='ttyS1'] size == 0",
}
# Enable & start the service. My version of puppet does not support upstarts on CentOS so I can't do this:
service { "ttyS1":
provider => upstart,
ensure => running,
enable => true,
}
# Instead I have created my own type
define upstart($ensure = "running", enable = "true") {
service { "upstart-$name":
provider => 'base',
ensure => $ensure,
enable => $enable,
hasstatus => true,
start => "/sbin/initctl start $name",
stop => "/sbin/initctl stop $name",
status => "/sbin/initctl status $name | /bin/grep -q '/running'",
}
}
upstart{"ttyS1": }
And finally, I need to make sure the server BIOS is configured with the VSP.
package { "hp-health": ensure => present } ->
service { "hp-health":
ensure => running,
hasstatus => true,
hasrestart => true,
enable => true,
} ->
exec { "vsp":
logoutput => "true",
path => ["/bin", "/sbin", "/usr/bin", "/usr/sbin"],
command => "/sbin/hpasmcli -s 'SET SERIAL VIRTUAL COM2'",
unless => "/sbin/hpasmcli -s 'SHOW SERIAL VIRTUAL' | grep 'The virtual serial port is currently COM2'",
require => Class['hp_drivers::service'],
} ->
# while we are messing with the serial ports, make COM1 work as the physical device
exec { "com1":
logoutput => "true",
path => ["/bin", "/sbin", "/usr/bin", "/usr/sbin"],
command => "/sbin/hpasmcli -s 'SET SERIAL EMBEDDED PORTA COM1'",
unless => "/sbin/hpasmcli -s 'SHOW SERIAL EMBEDDED' | grep 'Embedded serial port A: COM1'",
require => Class['hp_drivers::service'],
}
This sounded interesting so I though I would try it out. I was using this page as a guide http://www.fatmin.com/2011/06/redirect-linux-console-to-hp-ilo-via-ssh.html but it was a bit out of date.
The first thing to mention was that this does indeed work with iLO3. When configuring your iLO3, remember that in order to ssh to iLO3 you must have a PTY (-t -t), use protocol version 2 (-2) and use DSA keys (ssh-keygen -t dsa). My old configuration for older versions of iLO used exactly the opposite for all of these settings which took some time to figure out (and HP only recently identified/fixed the bugs with ssh keys).
Once you can ssh to your iLO interface, you can issue the command
VSP
and it will start a terminal emulator (actually, it just passes everything through to your terminal program so you can use xterm if you want but safer to assume a vt102). To exit, press ESCape and then ( and you will return to the iLO prompt.
For the OS configuration, I found that I had to make a few alterations. First, RHEL6/CentOS6 uses upstarts and not inittab. If you are not booting with a serial console you must create your own init script:
/etc/init/ttyS1.conf
# ttyS1 - agetty
#
# This service maintains a agetty on ttyS1 for iLO3 VSP.
stop on runlevel [S016]
start on runlevel [235]
respawn
exec agetty -8 -L -w /dev/ttyS1 115200 vt102
You can then enable the console with the command
start ttyS1
Next time you reboot it should start automatically.
In order to log in as root you must add the terminal it to the /etc/securetty file:
echo '/dev/ttyS1' >> /etc/securetty
So that was all good. Now I want to roll this out to all my servers. To do this I needed some puppet magic. Not surprising, puppet let me down on most of the steps for this.
# Install the init script. This one is easy, just dump in the file
file { "/etc/init/ttyS1.conf":
...
}
# Add the entry to securetty
augeas { "securetty_ttyS1":
# Ha ha. securetty has a special lens which is different to most other config files
context => "/files/etc/securetty",
changes => [ "ins 0 before /files/etc/securetty/1",
"set /files/etc/securetty/0 ttyS1",
],
onlyif => "match *[.='ttyS1'] size == 0",
}
# Enable & start the service. My version of puppet does not support upstarts on CentOS so I can't do this:
service { "ttyS1":
provider => upstart,
ensure => running,
enable => true,
}
# Instead I have created my own type
define upstart($ensure = "running", enable = "true") {
service { "upstart-$name":
provider => 'base',
ensure => $ensure,
enable => $enable,
hasstatus => true,
start => "/sbin/initctl start $name",
stop => "/sbin/initctl stop $name",
status => "/sbin/initctl status $name | /bin/grep -q '/running'",
}
}
upstart{"ttyS1": }
And finally, I need to make sure the server BIOS is configured with the VSP.
package { "hp-health": ensure => present } ->
service { "hp-health":
ensure => running,
hasstatus => true,
hasrestart => true,
enable => true,
} ->
exec { "vsp":
logoutput => "true",
path => ["/bin", "/sbin", "/usr/bin", "/usr/sbin"],
command => "/sbin/hpasmcli -s 'SET SERIAL VIRTUAL COM2'",
unless => "/sbin/hpasmcli -s 'SHOW SERIAL VIRTUAL' | grep 'The virtual serial port is currently COM2'",
require => Class['hp_drivers::service'],
} ->
# while we are messing with the serial ports, make COM1 work as the physical device
exec { "com1":
logoutput => "true",
path => ["/bin", "/sbin", "/usr/bin", "/usr/sbin"],
command => "/sbin/hpasmcli -s 'SET SERIAL EMBEDDED PORTA COM1'",
unless => "/sbin/hpasmcli -s 'SHOW SERIAL EMBEDDED' | grep 'Embedded serial port A: COM1'",
require => Class['hp_drivers::service'],
}
Subscribe to:
Posts (Atom)