FIPS integrity verification test failed.

After installing the updates for CentOS-6.6, my box started showing this warning when I run ssh:

FIPS integrity verification test failed.

I did not enable FIPS and don't much want it.
There are two ways to turn FIPS off. First option is to remove /etc/system-fips
rm -f /etc/system-fips
Second is to uninstall dracut-fips (which is the package which owns /etc/system-fips).
rpm --erase dracut-fips
The down side of this is that the package might come back. Something obviously caused it to be installed in the first place though I don't know what. There don't seem to be any rpm dependencies.

puppet augeas and sudo

I wanted to configure some sudo rules using puppet.


The default sudo config has an directory called /etc/sudoers.d which makes dropping in the actual entries rather easy:

file { "/etc/sudoers.d/example" :
        ensure => present,
        owner => 'root',
        group => 'root',
        mode => 0440,
        content => template('example/sudo.erb'),
}

but alas, the default RHEL6 sudo has requiretty set which prevented my sudo rules from working correctly.

Naturally I wanted to use augeas to remove that flag but it turned into a nightmare trifecta of puppet + augeas + sudo. Three tools with so much potential and a great lack of real world documentation.


I remember having battled with this before and giving up. This time I was determined to succeed. I revisited the only information on the internet but I still could not get it to work. After looking at the code for the sudo lens I was pretty sure that I had the correct version and eventually I was pointed in the right direction. Instead of removing the requiretty I needed to negate it. After some more mucking around I came up with a working incantation:

augeas { "turn off sudo requiretty":
        changes => [
                'set /files/etc/sudoers/Defaults[*]/requiretty/negate ""',
        ],
}

I hope that will be of use to someone.

ProLiant Virtual Serial Port

While working through the configuration of iLO3 for our HP DL380 G7 servers, I found a few pages talking about the Virtual Serial Port (VSP).

This sounded interesting so I though I would try it out. I was using this page as a guide http://www.fatmin.com/2011/06/redirect-linux-console-to-hp-ilo-via-ssh.html but it was a bit out of date.

The first thing to mention was that this does indeed work with iLO3. When configuring your iLO3, remember that in order to ssh to iLO3 you must have a PTY (-t -t), use protocol version 2 (-2) and use DSA keys (ssh-keygen -t dsa). My old configuration for older versions of iLO used exactly the opposite for all of these settings which took some time to figure out (and HP only recently identified/fixed the bugs with ssh keys).

Once you can ssh to your iLO interface, you can issue the command
VSP
and it will start a terminal emulator (actually, it just passes everything through to your terminal program so you can use xterm if you want but safer to assume a vt102). To exit, press ESCape and then ( and you will return to the iLO prompt.

For the OS configuration, I found that I had to make a few alterations. First, RHEL6/CentOS6 uses upstarts and not inittab. If you are not booting with a serial console you must create your own init script:
/etc/init/ttyS1.conf
# ttyS1 - agetty
#
# This service maintains a agetty on ttyS1 for iLO3 VSP.

stop on runlevel [S016]
start on runlevel [235]

respawn
exec agetty -8 -L -w /dev/ttyS1 115200 vt102

You can then enable the console with the command
start ttyS1
Next time you reboot it should start automatically.

In order to log in as root you must add the terminal it to the /etc/securetty file:
echo '/dev/ttyS1' >>  /etc/securetty

So that was all good. Now I want to roll this out to all my servers. To do this I needed some puppet magic. Not surprising, puppet let me down on most of the steps for this.
# Install the init script. This one is easy, just dump in the file 

file { "/etc/init/ttyS1.conf":
...
}

# Add the entry to securetty
augeas { "securetty_ttyS1":
# Ha ha. securetty has a special lens which is different to most other config files
context => "/files/etc/securetty",
changes => [ "ins 0 before /files/etc/securetty/1",
             "set /files/etc/securetty/0 ttyS1",
],
onlyif  => "match *[.='ttyS1'] size == 0",
}

# Enable & start the service. My version of puppet does not support upstarts on CentOS so I can't do this:
service { "ttyS1":
provider => upstart,
ensure => running,
enable => true,
}
# Instead I have created my own type
define upstart($ensure = "running", enable = "true") {
        service { "upstart-$name":
                provider => 'base',
                ensure => $ensure,
                enable => $enable,
                hasstatus => true,
                start  => "/sbin/initctl start $name",
                stop   => "/sbin/initctl stop $name",
                status => "/sbin/initctl status $name | /bin/grep -q '/running'",
        }
}
upstart{"ttyS1": }

And finally, I need to make sure the server BIOS is configured with the VSP.

package { "hp-health": ensure => present } ->
service { "hp-health":
        ensure => running,
        hasstatus => true,
        hasrestart => true,
        enable => true,
} ->
exec { "vsp":
        logoutput => "true",
        path => ["/bin", "/sbin", "/usr/bin", "/usr/sbin"],
        command => "/sbin/hpasmcli -s 'SET SERIAL VIRTUAL COM2'",
        unless => "/sbin/hpasmcli -s 'SHOW SERIAL VIRTUAL' | grep 'The virtual serial port is currently COM2'",
        require => Class['hp_drivers::service'],
} ->

# while we are messing with the serial ports, make COM1 work as the physical device
exec { "com1":
        logoutput => "true",
        path => ["/bin", "/sbin", "/usr/bin", "/usr/sbin"],
        command => "/sbin/hpasmcli -s 'SET SERIAL EMBEDDED PORTA COM1'",
        unless => "/sbin/hpasmcli -s 'SHOW SERIAL EMBEDDED' | grep 'Embedded serial port A: COM1'",
        require => Class['hp_drivers::service'],
}

Puppet tip

I have a custom puppet type called iptables. Not surprisingly it is used to build iptables rules.

Recently I was setting up a router which required MASQUERADING a few virtual network interfaces. I found that the module was lacking the ability to negate a match. For example I wanted a rule:
-A FORWARD ! -s 192.168.1.0/24 -i vnet -j DROP

It seemed like a simple task to add in the ability to insert the !. After all it is mostly just string manipulation.

My puppet setup already has pluginsync enabled and working so I though deploying my changes would be easy too.

I managed to make my changes (even though ruby is quite a foreign language to me). When I ran puppet I could see the changes being pulled down to the client but alas, it did not work correctly.

I checked with several web pages which try to explain how easy it is to add new properties and everything I was doing was correct.

After wasting hours tinkering in the ruby code I had the idea to restart the puppet master and bang, it started working.

Seems so obvious in hindsight.

Puppet DNS lookup

From time to time NetworkManager breaks my /etc/resolv.conf. I normally turn that off using puppet but when /etc/resolv.conf is broken, puppet won't run so I have to fix it manually :-(

Well, not any more. Now I add a static /etc/hosts entry for the puppet server and puppet will run even when /etc/resolv.conf is broken.

The special part about this is that I look up the IP address using a template so I don't have to hard code the value in my manifest. Google said it could not be done without writing a custom function. As always, I did not believe it...

Assuming you have set $puppetserver to contain the fqdn of your puppet server:

host { 'hostsconf-puppet':
        ensure => present,
        ip => inline_template("<% _erbout.concat(Resolv::DNS.open.getaddress('$puppetserver').to_s) %>"),
        name => $puppetserver,
        target => '/etc/hosts',
}

/usr/bin/ld: cannot find -lm

I was updating some of my software to run on CentOS6 and I had an unexpected error:

/usr/bin/ld: cannot find -lm
collect2: ld returned 1 exit status

The tool I was building uses -static and it turns out that static libraries are no longer shipped in glibc-devel. They are in a new package called glibc-static.

yum install glibc-static

solved the problem. My program now compiles and thanks to puppet, glibc-static is now installed on all my systems.
package { 'glibc-static': ensure => present }