Windows 8?

My first brush win WinPE turned out to be rather successful http://blog.chrysocome.net/2013/02/pxe-boot-winpe.html but recently our Windows team upgraded the SCCM server. Now when I attempt to install Windows 7 in KVM on CentOS-6, I get a windows 8 logo and then the dreaded error 0x0000005D.

The cause for this is, as always, long and complex. The new SCCM release now uses WinPE version 4 which is bases on Windows 8. Windows 8 requires a minimum level of CPU features to run. If you don't meet the minimum you get a well worded error message (well, at least it is easier to search for than a BSOD report).

I can't do much about SCCM, WinPE or Windows 8 so the next part of the problem is why does my KVM virtual machine not meet the Windows 8 requirements?

You guessed it. Bugs! It seems (more or less) that the 'sep' cpu feature was forgotten by libvirt and there is no fix coming soon.

What is needed then is a well implemented work around. KVM does support the required flag (-cpu +sep) but libvirt has no method to pass the flag to kvm. I already have a wrapper around kvm http://blog.chrysocome.net/2013/05/can-kvm-guest-found-out-who-its-host-is.html so it seemed logical to extend that script to add the missing flag.

Below is my solution which adds the +sep flag to the existing CPU configuration as well as set the serial number, as per the original script. Installation is the same as shown in my other blog post, edit the guest and set the <emulator> path to /usr/local/libexec/qemu-kvm (either using virsh edit or your favourite XML editor).

/usr/local/libexec/qemu-kvm
#!/bin/bash
# This is a wrapper around qemu which will supply
# DMI information
# and correct a bug with the CPU type required for winpe4 (Windows 8)
max=${#@}
index=0

for i in $(seq 1 $max) ; do
   p=${@:$i:1}
   if [ "$p" = "-cpu" ] ; then
      (( index = $i + 1 ))
      break
   fi
done

if [ $index -gt 0 ] ; then
cpu=${@:$index:1}
cpu="$cpu,+sep"

(( ibefore = $index - 1 ))
(( iafter = $index + 1 ))
set -- "${@:1:$ibefore}" $cpu "${@:$iafter}"
fi

if [ "$1" = "-name" ] ; then
    SERIAL=$(/usr/bin/hal-get-property --udi /org/freedesktop/Hal/devices/computer --key system.hardware.serial)
    exec /usr/libexec/qemu-kvm "$@" -smbios type=1,serial="KVM-$SERIAL"
else
    exec /usr/libexec/qemu-kvm "$@"
fi

Using xorg without a mouse

I always like it when I learn something new. Particularly when I read something one the web which turn out to work.

I wanted to find a way to move the mouse cursor without a mouse. I can actually do this with my lirc based remote but I have a new wireless keyboard for my linux 'TV' and I wanted to have a way to use that too.

A google search turned up an interesting page which suggested the X.org has this functionality built in and to my surprise it worked. It even had instructions for keyboards without a numeric keypad (like mine). Apparently "MouseKeys" and "PlotMode" has always been a feature in X.org and XFree86.

I must be in PlotMode at the moment which is quite slow. I will have to practice enabling Accelerated mode <Fn><Alt><Shift><Num Lock> should be the key I need.

puppet augeas and sudo

I wanted to configure some sudo rules using puppet.


The default sudo config has an directory called /etc/sudoers.d which makes dropping in the actual entries rather easy:

file { "/etc/sudoers.d/example" :
        ensure => present,
        owner => 'root',
        group => 'root',
        mode => 0440,
        content => template('example/sudo.erb'),
}

but alas, the default RHEL6 sudo has requiretty set which prevented my sudo rules from working correctly.

Naturally I wanted to use augeas to remove that flag but it turned into a nightmare trifecta of puppet + augeas + sudo. Three tools with so much potential and a great lack of real world documentation.


I remember having battled with this before and giving up. This time I was determined to succeed. I revisited the only information on the internet but I still could not get it to work. After looking at the code for the sudo lens I was pretty sure that I had the correct version and eventually I was pointed in the right direction. Instead of removing the requiretty I needed to negate it. After some more mucking around I came up with a working incantation:

augeas { "turn off sudo requiretty":
        changes => [
                'set /files/etc/sudoers/Defaults[*]/requiretty/negate ""',
        ],
}

I hope that will be of use to someone.

Sparse files on Windows

Once again I am drawn away from Linux to solve a Windows problem. The source of the problem is Hyper-V which (as always) has a cryptic error message about 'cannot open attachment' and 'incorrect file version'.

The source of the error was tracked down to the file being flagged as Sparse.

What is a sparse file?
Under UNIX/Linux, a sparse file is a file where not all of the storage for the file has been allocated. Handling of sparse files is normally transparent but some tools like file copy and backup programs can handle sparse files more efficiently if they know where the sparse bits are. Getting that information can be tricky.

In contrast, under windows, a sparse file is a file which has the sparse flag set. Presumably the sparse flag is set because not all of the storage for the file has been allocated (much like under Linux). Interestingly, even if all the storage is allocated, the sparse flag may still be set. (It seems the flag indicates the potential to be sparse rather than actually being sparse. There is an API to find out the actual sparse parts).

The the problem started when I happened to download a Hyper-V virtual machine using BitTorrent. When the files are being created, not all of the content exists so it is indeed sparse. Once all the content has been supplied, the file is (to my mind anyway) no longer sparse. However, under windows it seems, once a sparse file, always a sparse file.

Microsoft provide a tool to check and set the sparse flag:
fsutil sparse queryflag <filename>
fsutil sparse setflag <filename>
Note 1: Have they not heard of get and set
Note 2: You can't use a wildcard for <filename>
The amazing thing to note here is that there is no clearflag option. This might lead you to believe that you can not do that. In fact you can. For users in a pickle, there is a program called Far Manager which can (among other things) clear the flag. Far Manager is open source and a quick peek at the code shows that it uses a standard IOCTL to do this named FSCTL_SET_SPARSE.

So with that knowledge, it is actually quite easy to make a file not be sparse any more. In fact, I wrote a program called unsparse.

Not only does the tool have the ability to clear the sparse flag, it can recursively process a directory and unsparse all the sparse files found, making it perfect to fix up a Hyper-V download.

Look for the program soon on my chrysocome website http://www.chrysocome.net/unsparse

@reboot jobs will be run at computer's startup.

"@reboot jobs will be run at computer's startup."
What on earth does that mean?

These days RedHat use cronie as the system cron daemon. Described as 'based on the original cron' I think it is a fork of vixi-cron which was used until EL5.

For some time, both of these cron daemons have had an @reboot syntax which allows you to run scripts at (more or less) boot time (when the cron daemon is started). This allows users to start long running processes without the sysadmin having to write an initscript.

It also happens that from time to time, the cron daemon crashes. This is not ideal because cron is the very tool which can be used to periodically confirm that a daemon has not crashed. For now I have added a check to puppet to ensure that the cron daemon is running.

When the cron daemon is started, it logs some messages, one of which is the cryptic:
@reboot jobs will be run at computer's startup.
message. I understand that it is trying to tell me something but it fals short of conveying the message. The internet did not have much to say on the topic either so I had to resort to the source code.

The source of the message is from within the run_reboot_jobs function. The function first checks the existence of a (so called) lock file. The file is
/var/run/cron.reboot
If this file is present, the message is printed out and none of the @reboot jobs are run. If the file is absent, it is created and then the @reboot jobs are queued up to be run.

Perhaps the message should read:
Lock file /var/run/cron.reboot present. @reboot jobs have already been run. skipping.

So that is the mystery almost solved. The remaining details are that during boot, the rc.sysinit script removed a number of stale lock files, including the contents of /var/run. This ensures that at boot time, the cron daemon runs the @reboot jobs.

If you wanted to re-run the @reboot jobs without rebooting your server, you can easily trick it with:
rm /var/run/cron.reboot
service crond restart

Perhaps that could also be added to the init script so you run
service crond restart-boot

chkconfig priorities

I recently had a problem on a CentOS-6 machine where my dhcp server did not start at boot because it was serving a virtual network interface which did not yet exist.

The best solution to this problem would be for the dhcp server to start up and wait for the network interface to come up. Many other network tools do this successfully so I don't know why dhcpd should be different. That problem is however too big for me to fix on my server so I need a more simple approach.

To solve this, I would like to adjust my dhcp server to start after the virtual network interface service. CentOS-6 uses the (old) RedHat style init scripts (with some LSB configuration too). RedHat like to proclaim that they use upstarts now but all the upstarts do is make a call to the /etc/rc.d/rc script, just like init used to do.

So, the problem is still based around the System V style scripts which have magic comments which define when to start & stop the scripts.

This is the relevant parts of the header for the dhcp server (/etc/rc.d/init.d/dhcpd):
### BEGIN INIT INFO
# Provides: dhcpd
# Default-Start:
# Default-Stop:
# Should-Start: portreserve
# Required-Start: $network
# Required-Stop:
# Short-Description: Start and stop the DHCP server
# Description: dhcpd provides the Dynamic Host Configuration Protocol (DHCP)
#              server.
### END INIT INFO
#
# The fields below are left around for legacy tools (will remove later).
#
# chkconfig: - 65 35

Despite the comments about being legacy, when installed using chkconfig, the priorities used are indeed Start 65, Kill 35. We can confirm this with a simple ls
ls /etc/rc.d/rc?.d/*dhcpd

The virtual network is also controlled by an init script with a more modest header (/etc/rc.d/init.d/vand):
# chkconfig: 2345 95 05
# description: Virtual Area Network Deamon

Not surprisingly, this will Start at 95 and Kill at 05.

The first idea I had was to modify the dhcpd init scrip directly. Unfortunately this script is not marked as config file (in the rpm) so when a new version comes out, my changes will be lost. I need something better than that.

It is not immediately obvious but chkconfig does have the ability to alter the priorities. This is called override (but not to be confused with the command line parameter --override).


I was unable to find and documentation on the override but I did work out that his minimal config file was all that was needed:


/etc/chkconfig.d/dhcp:
### BEGIN INIT INFO
# Required-Start: vand
### END INIT INFO

When chkconfig reads the headers from the dhpcd init script, it will also read this file (because it has the same basename) and override the values from the init script.

All that is needed is to apply the settings with:
chkconfig dhcpd on

Some information about LSB init scripts can be found here http://refspecs.linuxbase.org/LSB_3.1.1/LSB-Core-generic/LSB-Core-generic/initscrcomconv.html

The final job is to make puppet aware of my changes. Perhaps in my next post.

What happened to java -client?

On our shared server, users are limited to 25 processes. A login script further reduces the soft limit by another 5 processes. This safety margin allows the user to log in an kill something which is using up all their processes. If they want they can raise their soft limit back up to the hard limit.

This works well for most things but java insists on using lots of processes. Confusingly, it prints this message:
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# An error report file with more information is saved as:
# ./hs_err_pid27297.log
The log file is not of much use. It repeats the claim about 'insufficient memory' which is not true and none of the listed solutions will help.

The problem is that the default garbage collector (GC) scales up with the number of CPU cores. Our old server had 8 but the new server has 24. This is easily pushing the users over the 20 thread soft limit.

I don't if the log file lists the number of java threads. If it does I can't work out how to read it. If it could be found it could be compared with the number of available processes:
expr $(ulimit -u) - $(ps x | wc --lines)

In the past I used to solve this problem by adding this to the login script:
alias java="java -client"

but that does not work any more. I don't know when it was removed but it leaves me with the headache of finding a new workaround.

After some research I found this page https://wiki.csiro.au/pages/viewpage.action?pageId=545034311 which suggested using
-XX:ParallelGCThreads=1 -XX:+UseParallelGC
or even just
-XX:+UseSerialGC

This seems to solve the immediate problem but still is not all that useful. To specify the flag to run java you need this syntax:

java -XX:+UseSerialGC

but for javac you have to specify like this:
javac -J-XX:+UseSerialGC


and I don't expect anyone should have to remember that. One simple solution is to alias both those commands but there are many other tools which are part of java which would also need aliases. These other tools may (and do) use their own unique command line parameters which I would rather not have to learn.


Luckily there is also an environment variable you can set like this:
export _JAVA_OPTIONS="-XX:ParallelGCThreads=1 -XX:+UseParallelGC"

however, that makes all the java programs print out this annoying message:
Picked up _JAVA_OPTIONS: -XX:ParallelGCThreads=1 -XX:+UseParallelGC
And the word on the internet is that there is no way to suppress that message.

My solution to use an alias for the most common tools java and javac while keeping the environment variable for all other programs. At first I tried to use this in my alias:
_JAVA_OPTIONS= /usr/bin/java -XX:+UseSerialGC
but an empty variable still shows the message (and best I can tell, bash won't unset a variable like that). I needed to unset it so I had to combine the env command like this:
env -u _JAVA_OPTIONS -XX:+UseSerialGC

My final profile script looks like this:
export _JAVA_OPTIONS="-XX:ParallelGCThreads=1 -XX:+UseParallelGC"
#export _JAVA_OPTIONS="-XX:+UseSerialGC"
alias java="env -u _JAVA_OPTIONS $(which --skip-alias java) $_JAVA_OPTIONS"
alias javac="env -u _JAVA_OPTIONS $(which --skip-alias javac)$(for i in $_JAVA_OPTIONS ; do echo -n " -J$i" ; done)"
You can select which GC you want. I have demonstrated with the more complex example which has two options to be prefixed for javac.

The java users can now happily compile away without java needing the entire machine resources for every invocation.