XBMC on CentOS 6

After years of putting it off I have finally taken the plunge into XBMC.

My plan is to hook it up as a front end for the Digital ORB pvr.

RpmFusion have packages for XBMC version 11 for CentOS 6. This is a good start as many of the audio & video packages from RpmForge (which I am sure most CentOS 6 users have) are too old for XBMC.

Unfortunately there is no package for XBMC version 12 which apparently has much better pvr support. The bug database entry suggested that it would be relatively easy to fix... and it was https://bugzilla.rpmfusion.org/show_bug.cgi?id=2699

So now I have a relatively stable base to start my integration. Lets hope that I get it working before XBMC version 13 comes out and changes everything again.

If you want to run XBMC yourself, you will need my xbmc and taglib packages from http://www.chrysocome.net/downloads and any dependencies from rpmfusion and epel.

You can also get the Digital ORB from my downloads directory. It does come with instructions but has not been widely tested yet. Stay tuned.

Large filesystem on CentOS-6

I had the pleasure of testing out a new server which came with 10 x 4Tb drives. Configured on and HP Smart Array P420 in RAID 50 that gave me a 32Tb drive to use.

I installed CentOS-6 but the installer would not allow me to use the full amount of space. It turns out the maximum ext4 filesystem on CentOS-6 is 16Tb. This is the limit of 32bit block addressing and 4k blocks. (2³² * 4k = 2³² * 2¹²= 2⁴⁴=2⁴⁰ * 2⁴ = 16T)

That did not make for a very exciting test. Only half the disk available was available in a single filesystem. Sure I could create two logical volumes and then create two filesystems but that seemed a bit like a DOS solution.

Some research turned up the options of using ext4 48bit block addressing. This enables a filesysem up to 1Eb in size and allows for future 64bit addressing for an even larger limit.

The catch was of course that 48bit addressing is not supported by the tools which come with CentOS-6.

Fedora 20 (rawhide) does come with the required updates to e2fsprogs (e2fsprogs-1.42.8-3) which enable 48bit addressing (somehow future-proofed by being called 64bit addressing).  I then set out to rebuild the fedora 20 rpm for CentOS-6. Amazingly the compile was clean though I did have to disable some tests which is a tad alarming. All in all it did not take long to produce a set of replacement rpms for e2fsprogs.

After installing the new tools I was able to create a new filesystem larger than 16Tb. I started at 17Tb to give me room to play. It seems that the CentOS-6 kernel does already have support for 48bit addressing. I ran a number of workloads and I could not find any problems. Still, I don't know what bugs may be lurking in there.

dumpe2fs shows the new filesystem feature '64bit'.

I also attempted to do an offline resize of the filesystem to the maximum size of my disk. This just worked as expected. Online resize is not available until a much later kernel version. I did not attempt this because there are know bugs.

The last limit I wanted to test was the file size limit. Even on my new filesystem the individual file size limit is 16Tb. It is not every day that I get to make one of them so I did
 dd if=/dev/zero of=big bs=1M count=$(expr 16 \* 1024 \* 1024)

At ~500Mb/s it still took 9 hours to complete.
And it turns out that the maximum file size is actually 17592186040320 and not 17592186044416. That is 4k (one block) short of 16Tb. (You can actually check that on any machine with the command truncate big --size 17592186040321 )

That also raised an issue which was a problem on ext3. How long does it take to delete a large file? Well this is the largest possible file and it took 13 minutes.

In conclusion, the 16Tb filesystem limit is easliy raised on CentOS-6 but it comes at the expense of using untested tools and kernel features. Although I did not find any problems in my testing, this could pose a substantial risk if you have over 16Tb of data which you do not want to loose.

If anyone is interested in my rpms you can get them from http://www.chrysocome.net/download

Windows 8?

My first brush win WinPE turned out to be rather successful http://blog.chrysocome.net/2013/02/pxe-boot-winpe.html but recently our Windows team upgraded the SCCM server. Now when I attempt to install Windows 7 in KVM on CentOS-6, I get a windows 8 logo and then the dreaded error 0x0000005D.

The cause for this is, as always, long and complex. The new SCCM release now uses WinPE version 4 which is bases on Windows 8. Windows 8 requires a minimum level of CPU features to run. If you don't meet the minimum you get a well worded error message (well, at least it is easier to search for than a BSOD report).

I can't do much about SCCM, WinPE or Windows 8 so the next part of the problem is why does my KVM virtual machine not meet the Windows 8 requirements?

You guessed it. Bugs! It seems (more or less) that the 'sep' cpu feature was forgotten by libvirt and there is no fix coming soon.

What is needed then is a well implemented work around. KVM does support the required flag (-cpu +sep) but libvirt has no method to pass the flag to kvm. I already have a wrapper around kvm http://blog.chrysocome.net/2013/05/can-kvm-guest-found-out-who-its-host-is.html so it seemed logical to extend that script to add the missing flag.

Below is my solution which adds the +sep flag to the existing CPU configuration as well as set the serial number, as per the original script. Installation is the same as shown in my other blog post, edit the guest and set the <emulator> path to /usr/local/libexec/qemu-kvm (either using virsh edit or your favourite XML editor).

/usr/local/libexec/qemu-kvm
#!/bin/bash
# This is a wrapper around qemu which will supply
# DMI information
# and correct a bug with the CPU type required for winpe4 (Windows 8)
max=${#@}
index=0

for i in $(seq 1 $max) ; do
   p=${@:$i:1}
   if [ "$p" = "-cpu" ] ; then
      (( index = $i + 1 ))
      break
   fi
done

if [ $index -gt 0 ] ; then
cpu=${@:$index:1}
cpu="$cpu,+sep"

(( ibefore = $index - 1 ))
(( iafter = $index + 1 ))
set -- "${@:1:$ibefore}" $cpu "${@:$iafter}"
fi

if [ "$1" = "-name" ] ; then
    SERIAL=$(/usr/bin/hal-get-property --udi /org/freedesktop/Hal/devices/computer --key system.hardware.serial)
    exec /usr/libexec/qemu-kvm "$@" -smbios type=1,serial="KVM-$SERIAL"
else
    exec /usr/libexec/qemu-kvm "$@"
fi

Using xorg without a mouse

I always like it when I learn something new. Particularly when I read something one the web which turn out to work.

I wanted to find a way to move the mouse cursor without a mouse. I can actually do this with my lirc based remote but I have a new wireless keyboard for my linux 'TV' and I wanted to have a way to use that too.

A google search turned up an interesting page which suggested the X.org has this functionality built in and to my surprise it worked. It even had instructions for keyboards without a numeric keypad (like mine). Apparently "MouseKeys" and "PlotMode" has always been a feature in X.org and XFree86.

I must be in PlotMode at the moment which is quite slow. I will have to practice enabling Accelerated mode <Fn><Alt><Shift><Num Lock> should be the key I need.

puppet augeas and sudo

I wanted to configure some sudo rules using puppet.


The default sudo config has an directory called /etc/sudoers.d which makes dropping in the actual entries rather easy:

file { "/etc/sudoers.d/example" :
        ensure => present,
        owner => 'root',
        group => 'root',
        mode => 0440,
        content => template('example/sudo.erb'),
}

but alas, the default RHEL6 sudo has requiretty set which prevented my sudo rules from working correctly.

Naturally I wanted to use augeas to remove that flag but it turned into a nightmare trifecta of puppet + augeas + sudo. Three tools with so much potential and a great lack of real world documentation.


I remember having battled with this before and giving up. This time I was determined to succeed. I revisited the only information on the internet but I still could not get it to work. After looking at the code for the sudo lens I was pretty sure that I had the correct version and eventually I was pointed in the right direction. Instead of removing the requiretty I needed to negate it. After some more mucking around I came up with a working incantation:

augeas { "turn off sudo requiretty":
        changes => [
                'set /files/etc/sudoers/Defaults[*]/requiretty/negate ""',
        ],
}

I hope that will be of use to someone.

Sparse files on Windows

Once again I am drawn away from Linux to solve a Windows problem. The source of the problem is Hyper-V which (as always) has a cryptic error message about 'cannot open attachment' and 'incorrect file version'.

The source of the error was tracked down to the file being flagged as Sparse.

What is a sparse file?
Under UNIX/Linux, a sparse file is a file where not all of the storage for the file has been allocated. Handling of sparse files is normally transparent but some tools like file copy and backup programs can handle sparse files more efficiently if they know where the sparse bits are. Getting that information can be tricky.

In contrast, under windows, a sparse file is a file which has the sparse flag set. Presumably the sparse flag is set because not all of the storage for the file has been allocated (much like under Linux). Interestingly, even if all the storage is allocated, the sparse flag may still be set. (It seems the flag indicates the potential to be sparse rather than actually being sparse. There is an API to find out the actual sparse parts).

The the problem started when I happened to download a Hyper-V virtual machine using BitTorrent. When the files are being created, not all of the content exists so it is indeed sparse. Once all the content has been supplied, the file is (to my mind anyway) no longer sparse. However, under windows it seems, once a sparse file, always a sparse file.

Microsoft provide a tool to check and set the sparse flag:
fsutil sparse queryflag <filename>
fsutil sparse setflag <filename>
Note 1: Have they not heard of get and set
Note 2: You can't use a wildcard for <filename>
The amazing thing to note here is that there is no clearflag option. This might lead you to believe that you can not do that. In fact you can. For users in a pickle, there is a program called Far Manager which can (among other things) clear the flag. Far Manager is open source and a quick peek at the code shows that it uses a standard IOCTL to do this named FSCTL_SET_SPARSE.

So with that knowledge, it is actually quite easy to make a file not be sparse any more. In fact, I wrote a program called unsparse.

Not only does the tool have the ability to clear the sparse flag, it can recursively process a directory and unsparse all the sparse files found, making it perfect to fix up a Hyper-V download.

Look for the program soon on my chrysocome website http://www.chrysocome.net/unsparse

@reboot jobs will be run at computer's startup.

"@reboot jobs will be run at computer's startup."
What on earth does that mean?

These days RedHat use cronie as the system cron daemon. Described as 'based on the original cron' I think it is a fork of vixi-cron which was used until EL5.

For some time, both of these cron daemons have had an @reboot syntax which allows you to run scripts at (more or less) boot time (when the cron daemon is started). This allows users to start long running processes without the sysadmin having to write an initscript.

It also happens that from time to time, the cron daemon crashes. This is not ideal because cron is the very tool which can be used to periodically confirm that a daemon has not crashed. For now I have added a check to puppet to ensure that the cron daemon is running.

When the cron daemon is started, it logs some messages, one of which is the cryptic:
@reboot jobs will be run at computer's startup.
message. I understand that it is trying to tell me something but it fals short of conveying the message. The internet did not have much to say on the topic either so I had to resort to the source code.

The source of the message is from within the run_reboot_jobs function. The function first checks the existence of a (so called) lock file. The file is
/var/run/cron.reboot
If this file is present, the message is printed out and none of the @reboot jobs are run. If the file is absent, it is created and then the @reboot jobs are queued up to be run.

Perhaps the message should read:
Lock file /var/run/cron.reboot present. @reboot jobs have already been run. skipping.

So that is the mystery almost solved. The remaining details are that during boot, the rc.sysinit script removed a number of stale lock files, including the contents of /var/run. This ensures that at boot time, the cron daemon runs the @reboot jobs.

If you wanted to re-run the @reboot jobs without rebooting your server, you can easily trick it with:
rm /var/run/cron.reboot
service crond restart

Perhaps that could also be added to the init script so you run
service crond restart-boot