Windows 8.1 in KVM

As Windows XP comes to its end of life, I have to get with the times and test my windows programs in Windows 8.1.

Naturally I am drawn to KVM which is where I run all my Windows instances.

Despite some initial concerns regarding the lack of the +sep flag (covered in my winpe pxe blog), installing Windows 8.1 turned out to be a breeze. I just used the settings for Windows 7 64bit.

After the install I was prompted to log. I have no interest in using a 'Microsoft Account' so I used the obscure procedure of 'Don't have an account' -> 'Sign in without a Microsoft account' which would be better described as 'Create local account'.

The next confronting issue was the start menu or whatever they call it these days. I worked out that I should not to look at it, just to start typing. That works more or less the same as the 'search' on Windows 7 (but possibly faster. It makes by mind boggle that it can take so long to run cmd.exe on Windows 7).

Once I had the basics under control, I set about installing the VirtIO & Spice drivers for extra performance and GUI features (like screen resizing). Alas, spice-guest-tools-0.74.exe does not support Windows 8 and fails during the install complaining about an unsupported version of Windows.

I know the virtio drivers work because I have used them in WinPE so I set about installing them manually. Although they are compatible, installing them is not straight forward.

The driver files are made available under Program Files (x86)\SPICE Guest Tools

The method I normally use is to use guestfish to install the driver file and configure it in the Critical Driver Data Base (CDDB) using a registry patch. Windows 8 no longer has a CDDB so this method does not work.

In the end I reverted to an old method:

• Add a new (temporary) VirtIO disk to the VM
• Boot into Windows
• Install the drivers for VirtIO
• Shut down windows
• Remove the temporary disk
• Change the main disk from IDE to VirtIO

This method works much like it has since Windows 7 allowed changing of the boot path.

The VirtIO network seems to work fine.

Talking to a Samba sever required adjusting some settings. First there are some updates which are apparently required (according to Microsoft) so I make sure all the OS patches were installed first.

Next I adjusted some policy settings under Administrative Tools -> Local Security Policy -> Local Policies -> Security Options:

• Network security: LAN Manager authentication level = Send LM & NTLM - use NTLMv2 session security if negotiated
• Network security: Minimum session security for NTLM SSP based (including secure RPC) clients = No Minimum

Then restart the workstation service or reboot.

Finally, the graphics drivers. For reasons which don't seem to be explained, the QXL video driver is not compatible with Windows 8.1. The performance of the 'Microsoft' driver is not too bad but it means no resizing which is annoying.

I am hoping that the spice people make their drivers and installer work soon.

One final observation is to do with the windows version number. Windows 8.1 is NT 6.3 but unless your program is manifested for windows 8.1, you will get the version number 6.2. This is by design. Some kind of new design which means the the GetVersion... API has been deprecated. What are they doing over at Microsoft???

Lazarus has come to life

Most of my windows programming is still done in Delphi 5. This was released in 1999 which makes it positively ancient. (Even more scary is that some programs started life in Delphi 2).

Not long after Delphi 5, Borland lost the plot (or prehaps they already had when they changed their name to Inprise). Although Delphi 7 was available, when I looked up upgrading the .net experiment was under way and it seemed like the end of the road for Delphi. I had hoped for a cross platform modern version to appear but that never really happened. Eventually the crazy was transferred to Embarcadero (whatever that is) and I still don't understand what works with what.


While all that was going on, I kept an eye on another program, Free Pascal. Free Pascal always showed such promise but the IDE was reminiscent of programming c++ under DOS.

Finally it seems though, I might have found what I have been looking for.  Lazarus, the Delphi styled IDE for Free Pascal. Matured enough for a 1.0 release and soon to be 1.2. Native unicode support and cross platform (Linux, Windows, OS-X).

Although there are a few sharp corners (don't import a unit called strings!), I am planning on converting my projects over to Lazarus and hopefully bringing them back from the dead too. My initial testing has been positive.

With any luck my programs will be playing nice with Windows 7, running on Linux and being recompiled without having to pay any license fees. It has taken a long time but I think it has been worth it.

Now if I could just press for Android support too...

Windows 8?

My first brush win WinPE turned out to be rather successful http://blog.chrysocome.net/2013/02/pxe-boot-winpe.html but recently our Windows team upgraded the SCCM server. Now when I attempt to install Windows 7 in KVM on CentOS-6, I get a windows 8 logo and then the dreaded error 0x0000005D.

The cause for this is, as always, long and complex. The new SCCM release now uses WinPE version 4 which is bases on Windows 8. Windows 8 requires a minimum level of CPU features to run. If you don't meet the minimum you get a well worded error message (well, at least it is easier to search for than a BSOD report).

I can't do much about SCCM, WinPE or Windows 8 so the next part of the problem is why does my KVM virtual machine not meet the Windows 8 requirements?

You guessed it. Bugs! It seems (more or less) that the 'sep' cpu feature was forgotten by libvirt and there is no fix coming soon.

What is needed then is a well implemented work around. KVM does support the required flag (-cpu +sep) but libvirt has no method to pass the flag to kvm. I already have a wrapper around kvm http://blog.chrysocome.net/2013/05/can-kvm-guest-found-out-who-its-host-is.html so it seemed logical to extend that script to add the missing flag.

Below is my solution which adds the +sep flag to the existing CPU configuration as well as set the serial number, as per the original script. Installation is the same as shown in my other blog post, edit the guest and set the <emulator> path to /usr/local/libexec/qemu-kvm (either using virsh edit or your favourite XML editor).

/usr/local/libexec/qemu-kvm
#!/bin/bash
# This is a wrapper around qemu which will supply
# DMI information
# and correct a bug with the CPU type required for winpe4 (Windows 8)
max=${#@}
index=0

for i in $(seq 1 $max) ; do
   p=${@:$i:1}
   if [ "$p" = "-cpu" ] ; then
      (( index = $i + 1 ))
      break
   fi
done

if [ $index -gt 0 ] ; then
cpu=${@:$index:1}
cpu="$cpu,+sep"

(( ibefore = $index - 1 ))
(( iafter = $index + 1 ))
set -- "${@:1:$ibefore}" $cpu "${@:$iafter}"
fi

if [ "$1" = "-name" ] ; then
    SERIAL=$(/usr/bin/hal-get-property --udi /org/freedesktop/Hal/devices/computer --key system.hardware.serial)
    exec /usr/libexec/qemu-kvm "$@" -smbios type=1,serial="KVM-$SERIAL"
else
    exec /usr/libexec/qemu-kvm "$@"
fi

Sparse files on Windows

Once again I am drawn away from Linux to solve a Windows problem. The source of the problem is Hyper-V which (as always) has a cryptic error message about 'cannot open attachment' and 'incorrect file version'.

The source of the error was tracked down to the file being flagged as Sparse.

What is a sparse file?
Under UNIX/Linux, a sparse file is a file where not all of the storage for the file has been allocated. Handling of sparse files is normally transparent but some tools like file copy and backup programs can handle sparse files more efficiently if they know where the sparse bits are. Getting that information can be tricky.

In contrast, under windows, a sparse file is a file which has the sparse flag set. Presumably the sparse flag is set because not all of the storage for the file has been allocated (much like under Linux). Interestingly, even if all the storage is allocated, the sparse flag may still be set. (It seems the flag indicates the potential to be sparse rather than actually being sparse. There is an API to find out the actual sparse parts).

The the problem started when I happened to download a Hyper-V virtual machine using BitTorrent. When the files are being created, not all of the content exists so it is indeed sparse. Once all the content has been supplied, the file is (to my mind anyway) no longer sparse. However, under windows it seems, once a sparse file, always a sparse file.

Microsoft provide a tool to check and set the sparse flag:
fsutil sparse queryflag <filename>
fsutil sparse setflag <filename>
Note 1: Have they not heard of get and set
Note 2: You can't use a wildcard for <filename>
The amazing thing to note here is that there is no clearflag option. This might lead you to believe that you can not do that. In fact you can. For users in a pickle, there is a program called Far Manager which can (among other things) clear the flag. Far Manager is open source and a quick peek at the code shows that it uses a standard IOCTL to do this named FSCTL_SET_SPARSE.

So with that knowledge, it is actually quite easy to make a file not be sparse any more. In fact, I wrote a program called unsparse.

Not only does the tool have the ability to clear the sparse flag, it can recursively process a directory and unsparse all the sparse files found, making it perfect to fix up a Hyper-V download.

Look for the program soon on my chrysocome website http://www.chrysocome.net/unsparse

Visual Studio 2010

I am not a fan of Visual Studio. Unfortunately I must use it for some projects. Recently I was forced to upgrade to Windows 7 and Visual Studio 2010. Not wanting to duplicate all my files, I decided to leave them on a network drive and just access them via the network.

Seems like a good idea, after all, why have a network if you store everything locally? Well, it seems that Visual Studio does not like that.

For some settings it will decide to find a suitable local directory for you. For some other settings it leaves you high and dry.

For example, when I try and build my program I get the error:

Error    1    error C1033: cannot open program database '\\server\share\working\project\debug\vc100.pdb'    \\server\share\working\project\stdafx.cpp    1    1    project

The internet was of little use which is why I thought I would put it in my blog.

This goes a long way to explain my criticism of Visual Studio. After installing several gig of software, is that the best error message it can come up with? Well I will try the help, Oh, that is online only. After jumping through some hoops, the help tells me that:

This error can be caused by disk error.

Well, no disks errors here. Perhaps it means that it does not like saving the .pdb on a network share. What is a .pdb anyway???

In the end, my solution (can I call it that or as Visual Studio hijacked that word?) was to save intermediate files locally:

• Open Project -> Properties...
• Select Configuration Properties\General
• Select Intermediate Directory
• Select <Edit...>
• Expand Macros>>
• Edit the value (by double clicking on the macros or just typing in):

$(TEMP)\$(ProjectName)\Debug\

• Select OK
• Select OK

And that seems to sort it out.

PXE boot WINPE

As part of our new Windows 7/AD deployment, SCCM is being used to control the imaging process of desktop computers.

We already have a comprehensive set of PXE enabled boot options so we needed a way to integrate the SCCM tools into our existing PXE setup.

The existing setup is syslinux(pxelinux) 3.11 on CentOS 5, ISC dhcpd and tftp.

We have several Linux live CDs, memtest, novell tools, DOS and chain booting to another (novell) server.

Our Microsoft team provide a bootable CD which we can use to image desktop machines and on some subnets they now provide the ability to PXE boot. Initially I hoped we could just chain boot to the SCCM server but that does not work.

After much research and testing I worked out a process using wimboot. Based on instructions found on http://ipxe.org/howto/sccm and http://forum.ipxe.org/showthread.php?tid=5745 I managed to write a script which converts a CD (ISO image).

The pxelinux entry is:
LABEL ad
com32 ad/linux.c32
append ad/wimboot initrd=ad/winpe.cpio

The files are in a tftp subdirectory called ad

linux.c32 is part of syslinux. I am using version 4.02 which was copied from a CentOS 6 server.

wimboot is available for download from ipxe.

winpe.cpio is a file we are going to create using my script below.

I also required a copy of bootmgr.exe which should be available on your microsoft tftp server but I was unable to find it and in the end I got a copy from our Mircosoft team.

Finally, I needed a copy of wimlib which has a linux version of the imagex tool. I was unable to find a RPM of this package and I just built it from source using the --without-ntfs-3g. (The source comes with .spec files so an RPM should be easy to build).

So here is my script:
convert_cd_into_pxe.sh:
#!/bin/bash
# Written by John Newbigin jnewbigin@chrysocome.net
ISO=x86_PROD_MS.iso

MNTPNT=wim
SCCMFILES=iso

IMAGEX=./wimlib-1.2.5/imagex

unalias cp

if [ "$(whoami)" = "root" ] ; then

   if [ ! -f bootmgr.exe ] ; then
      echo "You need a copy of bootmgr.exe"
      exit 1
   fi
   umount $SCCMFILES
   mkdir $SCCMFILES
   mkdir $MNTPNT
   mount -o loop $ISO $SCCMFILES

   cp $SCCMFILES/boot/bcd BCD
   cp $SCCMFILES/boot/boot.sdi .
   cp $SCCMFILES/sources/boot.wim .


   $IMAGEX mountrw boot.wim 1 $MNTPNT

   cp -drv $SCCMFILES/sms/* $MNTPNT/sms/
   umount $SCCMFILES
   rmdir $SCCMFILES

   ARCH=$(grep TsBootShell.exe $MNTPNT/Windows/System32/winpeshl.ini | cut -d \\ -f 4)
   # edit winpeshl.ini
   perl -pi -e "s|.*$ARCH.*|\"wscript.exe\",\"%SYSTEMDRIVE%\\\\sms\\\\bin\\\\$ARCH\\\\bootstrap.vbs\"|" $MNTPNT/Windows/System32/winpeshl.ini

   # install bootstrap.vbs
   cat > $MNTPNT/sms/bin/$ARCH/bootstrap.vbs << END
Set os = WScript.CreateObject ( "WScript.Shell" )
os.Run "%COMSPEC%", 7, false
os.Run "%COMSPEC% /c title Initialising... && wpeinit " & _
"&& net start dnscache", 1, true
os.Run WScript.ScriptFullName & "\..\TsmBootStrap.exe /env:WinPE " & _
"/configpath:%SYSTEMDRIVE%\sms\data", 1, true
END

   $IMAGEX unmount $MNTPNT --commit
   rmdir $MNTPNT

   ls BCD bootmgr.exe boot.sdi boot.wim | cpio --create -H newc > winpe.cpio
   rm BCD boot.sdi boot.wim
  
else
   echo "You must be root to run this script"
   exit 1
fi

I hope this is of use to someone.

Samba guest access

I was trying to share some photos for my wife to download from my linux desktop machine (CentOS 6). I had ~10Gig of photos but she only wanted a hand full. I though the best option would be to share out the folder using samba and she can use windows explorer to pick the ones she wants

Well it seems simple now but it took a few tries to get it working.

I use security=server which is not the most secure method but it is normally easy & convenient. It also seems to be poorly documented, particularly for more recent samba releases and versions of Windows.

My wife has an account on the password server but not on my new desktop. Guest access I though would be a simple solution here but no so. The problem is that guest access will not work by default when using security=server (despite what the man page says). There is a new setting called map to guest which defaults to Never. I had to change this to Bad User to get it to work.

There are the relevant parts from my working smb.conf:

security = server
password server = myserver

map to guest = Bad User

[photos]
comment = John's photos
path = /home/jnewbigin/photos
guest ok = yes
writable = no
force user = jnewbigin

So now it is working and I am happy and my wife is happy too.

Virtual Volumes and Dokan

It has been a holy grail for many years now and it is very close. The ability to assign a drive letter to your linux filesystems under Windows.

I am doing some final tests on my first release of Virtual Volumes which includes support for Dokan. Dokan is the Windows equivalent of FUSE under Linux. This means you can implement a filesystem in a userspace application.

There are some knows issues but my Windows XP install test was successful and I am currently doing some large file testing. If all goes well I might get a beta version released this weekend.