PHP Search Speed

I recently had a problem with a PHP command line script I use to sanity check some of our systems.

One particular check compares user accounts across a number of systems. There are ~100,000 user accounts. The script loads data in to memory from various LDAP and SQL interfaces. Once in memory, it iterates through the records and compares them. This process takes a few minutes, mostly spent loading data via LDAP.

I had identified a data problem in one system which uses SQL to get the user data. The SQL in question already joins a few tables and rather than introduce another complex join, I decided to just load the data about the new item into a new array using a separate SQL statement. This added just a few seconds to the script run time. However, once I implemented the record compare, the speed of the script slowed to a halt.

This was unexpected because I regularly process data sets this size and larger in PHP without and problems (once I raise the memory_limit).

This is what I discovered while debugging this problem:

It is well known that the PHP in_array & array_search functions are slow. They perform a linear search on the array resulting in O(n) performance.

Many people have contributed code which can implement a binary search on an array. This in theory reduces the performance to O(log n). My testing shows an improvement in search times but they are still quite slow and not representitive of the perforamce I am use to.
So how was it that I have never had this problem before? A quick inspection of my existing code shows that I always retrieve data by using the key. This results in a hashtable lookup of O(1).
So I changed my code to store my new data in the array key and use isset($array[$key]) and bang! my script is running on time again.

Footnote:

The performance of these functions may not be exactly O(n), O(log n) and O(1) but are close enough to enable comparing the performance of these functions against each other.

Eclipse software install error

I recently had the need to use Eclipse. I am not a fan and when I got this error while trying to install Eclipse plugin:

“Comparison method violates its general contract!”

my opinion was cemented.

The workaround is to use an old version of java like this:

eclipse -vm /usr/java/jdk1.6.0_45/bin/java


And the plugins, well they did not do much for me either.

Arduino IR Blaster

My IR blaster is built on a LeoStick which is an arduino compatible USB stick. I must be a sucker for punishment because I thought it would be easy to make one of the many IR libraries work on here and control my Samsung TV.

There were two problems with my plan. First, the defacto IR libraries don't allow me to use the necessary IO pins and second, Samsung IR codes are not supported.

Rather than detail what when wrong, I will present my solutions.

The first problem arises because the LeoStick has a green LED attached to Digital IO pin 9 which is the default PWM Timer used by Ken Shirriff's IR library. The solution is an updated fork of this library Chris Young which implements more Timer chips and enables selecting Digital IO pin 5. Do to that, edit IRLibTimer.h and find the section with the heading 'it's probably Leonardo' (the LeoStick is apparently compatible with the Leonardo). Comment out the default #define and uncomment the IR_USE_TIMER3. (I wonder why we can't select this at run time?). Based on the IRLib manual, I also installed my IR LED with whatever resistors I had on hand (470 ohm I think).

So, thanks to the hard work of these guys, I have got my LED working. Now, to make it talk Samsung protocol.

There were a few snippits of code in various forums but I had to piece that together into the format for IRLib. I have only added sending as I already have a list of codes from SamyGO and LIRC. The only code I wanted to use is TV_POWER 0xE0E040BF because once the TV is on, I can talk to it via the network.

I have submitted my changes as a GitHub pull request. If they are not accepted in the upstream library you can find them here https://github.com/jnewbigin/IRLib

The final piece to the puzzle is making the IR code send when I want it to. To keep this generic, I made my sketch read a hexadecimal code from the serial port. When it receives a code it just send it, thus allowing control over the time and the code. This could probably be integrated with LIRC but I am just using a simple shell command
echo 0xE0E040BF > /dev/ttyACM0

So here is my sketch. There is not much error detection in the serial reading but what is the worst that can go wrong?

/* Example program for from IRLib – an Arduino library for infrared encoding and decoding
 * Version 1.3   January 2014
 * Copyright 2014 by Chris Young http://cyborg5.com
 * Based on original example sketch for IRremote library
 * Version 0.11 September, 2009
 * Copyright 2009 Ken Shirriff
 * http://www.righto.com/
 *
 * Modified by John Newbigin to support Samsung and USB control May 2014
 */
#include <IRLib.h>

IRsend My_Sender;

void setup()
{
  Serial.begin(9600);
}

void p(char *fmt, ... ){
        char buf[128]; // resulting string limited to 128 chars
        va_list args;
        va_start (args, fmt );
        vsnprintf(buf, 128, fmt, args);
        va_end (args);
        Serial.print(buf);
}

// 0xE0E040BF = Power
void sendCode(unsigned long code)
{
   p("Sending SAMSUNG 0x%08lx\n", code);

  for (int i = 0; i < 3; i++)
  {
    My_Sender.send(SAMSUNG, code, 0);
    delay(30);
  }
}

unsigned long ir_code = 0;
void loop() {
  while(Serial.available() <= 0)
  {
    delay(10);
  }
  int incomingByte = Serial.read();

  int nibble = -1;
  if(incomingByte == 10)
  {
    p("Got newline\n");
    sendCode(ir_code);
    ir_code = 0;
  } 
  else if(incomingByte == 13)
  {
    // ignore
  }
  else if(incomingByte >= '0' && incomingByte <= '9')
  {
    nibble = incomingByte - '0';
    p("Got digit %d\n", nibble);
  }
  else if(incomingByte >= 'a' && incomingByte <= 'f')
  {
    nibble = incomingByte - 'a' + 10;
    p("Got hex digit %d\n", nibble);
  }
  else if(incomingByte >= 'A' && incomingByte <= 'F')
  {
    nibble = incomingByte - 'A' + 10;
    p("Got hex digit %d\n", nibble);
  }
  else if(incomingByte = 'x')
  {
    p("Starting number\n");
    ir_code = 0;
  }
  else
  {
    p("Got %d\n", incomingByte);
  }
 
  if(nibble >= 0)
  {
    ir_code<<= 4;
    ir_code+= nibble;
    p("Code is 0x%08lx\n", ir_code);
  }
}

Arduino on CentOS-6

I decided it was high time that I found a way to turn on my new TV automatically. After much research I decided that an arduino based IR Blaster was the way to go. I happen to have a Freetronics LeoStick which was given out a LCA in Ballarat a few years ago.

I recall at the time I could not get the arduino software to work on Linux (CentOS-5) so I resorted to using a Windows VM with USB passthru.

That was not going to cut it this time. I was going to use my full stubbornness to get it working on CentOS-6.

First of all, I had to get a copy of the IDE in RPM format. There was talk on the internet about just such a thing but I could not find an EL6 version. In the end I selected the Fedora 19 version which is suitable. The files needed are:

• arduino-1.0.5-6.fc19.noarch.rpm
• arduino-doc-1.0.5-6.fc19.noarch.rpm
• arduino-core-1.0.5-6.fc19.noarch.rpm

There are also a number of other dependancies because the IDE does not actually do much. Luckily the other bits are all in CentOS-6 or EPEL so using yum you can get it installed without too much trouble.

Running the IDE however is rigged with rubbish about adding your user account to certain groups. This seems antiquated to me and I refuse to do it. There must be a better way.

An ideal solution would be that the USB devices are assigned to the console owner and it would just work like any other peripheral. The way to implement that changes with every version of RHEL/CentOS but after some sleuthing, I found the answer for version 6 and it is no too difficult.

To get this working there are four jobs:

• Make sure the USB device ID is known to the USB TTY driver
• Set the appropriate permissions with udev
• Remove the restrictions on locking the device
• Configure the IDE

The first job is not immediately obvious and may not be applicable on all versions of Linux. On CentOS-6 (6.5) is was necessary. As if by magic or just infuriating design, the USB device ID of your arduino can change depending on what it is doing. This can be exploited for good, say if you wanted to build a USB keyboard or mouse but if you are using the default serial emulation then it seems unnecessary. For me, the default id used by the boot loader (26ba:0002) is automatically recognised as a serial device and /dev/ttyACM0 is created (by the cdc_acm driver) but when my sketch is running, the id (26ba:8002) is not recognised. This means the device goes away which confuses the arduino tools (and me). This is easily fixed by running this command as root:
echo "0x26ba 0x8002" >> /sys/bus/usb/drivers/cdc_acm/new_id
For your board, the IDs may well be different so you much check with dmesg or lsusb.

This needs to be done after every reboot so it should be added to an init script
modprobe cdc_acm
echo "0x26ba 0x8002" >> /sys/bus/usb/drivers/cdc_acm/new_id
TODO: is there is proper place for this? modprobe.conf? /etc/sysconfig/modules/arduino? For now, /etc/rc.d/rc.local will do.

Device permissions on CentOS-6 are managed by udev. Most devices use the traditional user, group and permissions. However, devices which are shared (for example, with console users) are managed using ACLs. (check with ls -l /dev/ | grep +).

In order to set the permissions, the correct devices must be identified. For me, the device name is /dev/ttyACM0 but this does not hang round for long (remember about the changing ids?). It is only present while the arduino bootloader is running. dmesg can give you a hint but the easiest way to check the device name is the plug in the stick and immediately run ls -l /dev/tty*

Once you know the device name you can interigate it with udev
udevadm info -a -n /dev/ttyACM0
Again you must run this immediatly after plugging in the USB stick.
It will output many properties of the device from which you must find something unique. At first I though that 'ATTRS{manufacturer}=="Freetronics"' was a good value to check for but it turns out that when your sketch is running, that string is not present. Better is to use the USB vendor ID 'ATTRS{idVendor}=="26ba"'.  While I was at it, I also added the Arduino vendor code of 20a0. I could then make a new udev rule by creating the file:
/etc/udev/rules.d/52-arduino.rules
and adding the contents:
ATTRS{idVendor}=="26ba", ENV{ACL_MANAGE}="1"ATTRS{idVendor}=="20a0", ENV{ACL_MANAGE}="1"

Some sources also recommend preventing the Modem Manager from probing the device. You can do that by adding this line too

ATTRS{idVendor}=="26ba", ENV{ID_MM_DEVICE_IGNORE}="1"
ATTRS{idVendor}=="20a0", ENV{ID_MM_DEVICE_IGNORE}="1"
Once the file is in place, you can force the rules to be applies with the command
udevadm trigger

Now check the device with ls -l again (immediately after plugging in the stick) and it should look like this (with a + sign)
crw-rw----+ 1 root dialout 166, 0 Apr 24 23:08 /dev/ttyACM0

If so, check the ACL with  getfacl /dev/ttyACM0
getfacl: Removing leading '/' from absolute path names
# file: dev/ttyACM0
# owner: root
# group: dialout
user::rw-
user:jnewbigin:rw-
group::rw-
mask::rw-
other::---

Note that my username jnewbigin is granted access. You should of course see your username there. If so, job done.

The next headache is device locking. The default rxtx library wants to write to a directory it does not have permissions on. Fixing this was painful. All I wanted was a simple way to skip the locking phase in the rxtx library. I had hoped that there would be an environment variable which could tune the locking. According to the documentation, the only way to turn locking off is by recompiling. Luckily the srpm is easily modified. You can download my lock free version of rxtx from http://www.chrysocome.net/download

Almost done. I would have though that configuring the IDE should not be required but as it happens, you must edit ~/.arduino/preferences.txt and change
serial.port=/dev/ttyACM0

So now we have removed the requirements to add your user into the unnecessary groups. The script which runs the IDE (arduino) however still complains that you are not a member. The best solution would be to rebuild the rpm without that check (and add in the udev rules) but for now you can either select Ignore or edit /usr/bin/arduino and change the line
for group in dialout lock; do
to
for group in ; do

Perhaps I will repackage it with these changes.


And stay tuned for the working IR blaster code!

OpenSCAD for CentOS 6

I am thinking about getting a 3D Printer but before jumping in, I wanted to test out some software to see how hard it is to drive these things.

OpenSCAD

It was suggested that I check out OpenSCAD, the programmers 3D CAD program.

As usual, for CentOS-6, it did not exist. Building it was tricky and required a bit of finesse. You can download my beta build and dependencies from http://www.chrysocome.net/download

Installation Instructions

Note: gmp 5.0 is named gmp5 so it can co-exist with the system version 4.3
Note: mpfr 3.1 is names mpfr3 so it can co-exist with the system version 2.4
yum install gmp5-5.0.5-3.el6.x86_64.rpm mpfr3-3.1.0-2.el6.x86_64.rpm CGAL-4.1-1.el6.x86_64.rpm opencsg-1.3.2-6.el6.x86_64.rpm openscad-2013.06-8.el6.x86_64.rpm --enablerepo=epel

I also need to make sure that the MCAD libraries are correctly packaged. I'll try and make an updated version soon.

FreeCAD

 

FreeCAD is a more traditional click and draw interface which is apparently very good and has many features. It can also be scripted with python. RPMs are available from RPM Fusion Nonfree

Conclusion

I need a lot of practice before I will be making any 3D objects!

musescore - sheet music on linux

Recently I was helping a friend who had written a song. We wanted to produce some sheet music so I though it would be worth checking out any open source music authoring software.

Many years ago I looked at software in this space and some promised much but in the end, none seemed to deliver on any front. At that time, shareware windows software was all that could be found.

This search was a lot more successful. It did not take me long to find MuseScore. This is a cross platform, open source sheet music editor (with playback). The windows version worked out of the box and we soon had the basics mastered; chords, notes, repeats, lyrics etc.

My next instinct was to get it to work on Linux, specifically CentOS-6. After installing a few requirements from rpmforge, I had the code compiled and running (with just a few minor bugs). Once I know that it worked, I wanted to make an RPM. Unfortunately, musescore (or mscore as it is called under Linux) uses an unusual build procedure based on cmake. Adding that into an RPM seemed a little daunting so I found the closest thing I could on the internet to use as a starting point.

Opensuse has an RPM but it would not build on CentOS. After a few simple tweaks I had the package building but it was not being packaged correctly. The %makeinstall macro was trying to install files into the filesystem and not the package buildroot. After much internet searching and testing, I found the answer (which is quite simple. So simple I don't know why no one had written it down before).

The same as you might use for a normal Makefile, just set the DESTDIR environment variable. My %install now has:
%install
export DESTDIR=%{buildroot}
cd build && %makeinstall
And with a few more minor tweaks I had the RPM finished.

Now I just have a few issues to sort out with playback on Linux. musescore does not like the default ALSA settings or ALSA devices (like my USB headset) disappearing and reappearing when I suspend and resume. Portaudio might fix that but it does not seem to work on my home machine.  The topic for another blog no doubt.

Lazarus has come to life

Most of my windows programming is still done in Delphi 5. This was released in 1999 which makes it positively ancient. (Even more scary is that some programs started life in Delphi 2).

Not long after Delphi 5, Borland lost the plot (or prehaps they already had when they changed their name to Inprise). Although Delphi 7 was available, when I looked up upgrading the .net experiment was under way and it seemed like the end of the road for Delphi. I had hoped for a cross platform modern version to appear but that never really happened. Eventually the crazy was transferred to Embarcadero (whatever that is) and I still don't understand what works with what.


While all that was going on, I kept an eye on another program, Free Pascal. Free Pascal always showed such promise but the IDE was reminiscent of programming c++ under DOS.

Finally it seems though, I might have found what I have been looking for.  Lazarus, the Delphi styled IDE for Free Pascal. Matured enough for a 1.0 release and soon to be 1.2. Native unicode support and cross platform (Linux, Windows, OS-X).

Although there are a few sharp corners (don't import a unit called strings!), I am planning on converting my projects over to Lazarus and hopefully bringing them back from the dead too. My initial testing has been positive.

With any luck my programs will be playing nice with Windows 7, running on Linux and being recompiled without having to pay any license fees. It has taken a long time but I think it has been worth it.

Now if I could just press for Android support too...

XBMC on CentOS 6

After years of putting it off I have finally taken the plunge into XBMC.

My plan is to hook it up as a front end for the Digital ORB pvr.

RpmFusion have packages for XBMC version 11 for CentOS 6. This is a good start as many of the audio & video packages from RpmForge (which I am sure most CentOS 6 users have) are too old for XBMC.

Unfortunately there is no package for XBMC version 12 which apparently has much better pvr support. The bug database entry suggested that it would be relatively easy to fix... and it was https://bugzilla.rpmfusion.org/show_bug.cgi?id=2699

So now I have a relatively stable base to start my integration. Lets hope that I get it working before XBMC version 13 comes out and changes everything again.

If you want to run XBMC yourself, you will need my xbmc and taglib packages from http://www.chrysocome.net/downloads and any dependencies from rpmfusion and epel.

You can also get the Digital ORB from my downloads directory. It does come with instructions but has not been widely tested yet. Stay tuned.

Sparse files on Windows

Once again I am drawn away from Linux to solve a Windows problem. The source of the problem is Hyper-V which (as always) has a cryptic error message about 'cannot open attachment' and 'incorrect file version'.

The source of the error was tracked down to the file being flagged as Sparse.

What is a sparse file?
Under UNIX/Linux, a sparse file is a file where not all of the storage for the file has been allocated. Handling of sparse files is normally transparent but some tools like file copy and backup programs can handle sparse files more efficiently if they know where the sparse bits are. Getting that information can be tricky.

In contrast, under windows, a sparse file is a file which has the sparse flag set. Presumably the sparse flag is set because not all of the storage for the file has been allocated (much like under Linux). Interestingly, even if all the storage is allocated, the sparse flag may still be set. (It seems the flag indicates the potential to be sparse rather than actually being sparse. There is an API to find out the actual sparse parts).

The the problem started when I happened to download a Hyper-V virtual machine using BitTorrent. When the files are being created, not all of the content exists so it is indeed sparse. Once all the content has been supplied, the file is (to my mind anyway) no longer sparse. However, under windows it seems, once a sparse file, always a sparse file.

Microsoft provide a tool to check and set the sparse flag:
fsutil sparse queryflag <filename>
fsutil sparse setflag <filename>
Note 1: Have they not heard of get and set
Note 2: You can't use a wildcard for <filename>
The amazing thing to note here is that there is no clearflag option. This might lead you to believe that you can not do that. In fact you can. For users in a pickle, there is a program called Far Manager which can (among other things) clear the flag. Far Manager is open source and a quick peek at the code shows that it uses a standard IOCTL to do this named FSCTL_SET_SPARSE.

So with that knowledge, it is actually quite easy to make a file not be sparse any more. In fact, I wrote a program called unsparse.

Not only does the tool have the ability to clear the sparse flag, it can recursively process a directory and unsparse all the sparse files found, making it perfect to fix up a Hyper-V download.

Look for the program soon on my chrysocome website http://www.chrysocome.net/unsparse

Visual Studio 2010

I am not a fan of Visual Studio. Unfortunately I must use it for some projects. Recently I was forced to upgrade to Windows 7 and Visual Studio 2010. Not wanting to duplicate all my files, I decided to leave them on a network drive and just access them via the network.

Seems like a good idea, after all, why have a network if you store everything locally? Well, it seems that Visual Studio does not like that.

For some settings it will decide to find a suitable local directory for you. For some other settings it leaves you high and dry.

For example, when I try and build my program I get the error:

Error    1    error C1033: cannot open program database '\\server\share\working\project\debug\vc100.pdb'    \\server\share\working\project\stdafx.cpp    1    1    project

The internet was of little use which is why I thought I would put it in my blog.

This goes a long way to explain my criticism of Visual Studio. After installing several gig of software, is that the best error message it can come up with? Well I will try the help, Oh, that is online only. After jumping through some hoops, the help tells me that:

This error can be caused by disk error.

Well, no disks errors here. Perhaps it means that it does not like saving the .pdb on a network share. What is a .pdb anyway???

In the end, my solution (can I call it that or as Visual Studio hijacked that word?) was to save intermediate files locally:

• Open Project -> Properties...
• Select Configuration Properties\General
• Select Intermediate Directory
• Select <Edit...>
• Expand Macros>>
• Edit the value (by double clicking on the macros or just typing in):

$(TEMP)\$(ProjectName)\Debug\

• Select OK
• Select OK

And that seems to sort it out.

Graphing the inputs of PCF8591

As discussed in previous posts, the PCF8591 provides up to four analog inputs. Displaying these inputs as numbers makes it hard to visualise what is really going on so here I will present some code which can graph the values in real time.

I am still using my demo board from I2C Analog to Digital Converter. (Thanks to Martin X for finding the YL-40 the schematic on the The BrainFyre Blog)

DX pcf8591-8-bit-a-d-d-a-converter-module-150190 YL-40 schematic

It is not clear from the schematic (or looking at the board) but the four inputs are:

• AIN0 - Jumper P5 - Light Dependent Resistor (LDR)
• AIN1 - Jumper P4 - Thermistor
• AIN2 - Not connected
• AIN3 - Jumper P6 - Potentiometer

It also seems that the board has I2C pull-up resistors which are not required because the Raspberry Pi already has them. This does not appear to cause any problems.

So my plan is to graph the four inputs so I can visualise them responding to changes.

Once again, I don't want to set out to teach C programming but I will be introducing a library called curses (actually ncurses) which makes it easy to display a text interface in a terminal window (virtual terminal).

It will also be able to adjust the analog output value using the + and - keys.

I will only add notes where I am doing something new from the example shown in Programming I2C.

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/ioctl.h>
#include <linux/i2c-dev.h>
We need a new header file
#include <ncurses.h>

int main( int argc, char **argv )
{
        int i;
        int r;
        int fd;
        unsigned char command[2];
        unsigned char value[4];
        useconds_t delay = 2000;

        char *dev = "/dev/i2c-1";
        int addr = 0x48;

        int j;
        int key;
Here we do some ncurses setup. This will allow us to check for a keyboard key press without having to wait for one to be pressed.
        initscr();
        noecho();
        cbreak();
        nodelay(stdscr, true);
        curs_set(0);

This will print out a message on the screen (at the current cursor location which is the top left)
        printw("PCF8591");


This will print out some labels for our graph bars. The text is printed at the specified location (row, column)
        mvaddstr(10, 0, "Brightness");
        mvaddstr(12, 0, "Temperature");
        mvaddstr(14, 0, "?");
        mvaddstr(16, 0, "Resistor");

We must now call refresh which will cause ncurses to update the screen with our changes
        refresh();
        fd = open(dev, O_RDWR );
        if(fd < 0)
        {
                perror("Opening i2c device node\n");
                return 1;
        }

        r = ioctl(fd, I2C_SLAVE, addr);
        if(r < 0)
        {
                perror("Selecting i2c device\n");
        }

        command[1] = 0;
        while(1)
        {
                for(i = 0; i < 4; i++)
                {
                        command[0] = 0x40 | ((i + 1) & 0x03); // output enable | read input i
                        r = write(fd, &command, 2);
                        usleep(delay);
                        // the read is always one step behind the selected input
                        r = read(fd, &value[i], 1);
                        if(r != 1)
                        {
                                perror("reading i2c device\n");
                        }
                        usleep(delay);
 The full range of the analog value 0 - 255 would not fit on most screens so we scale down by a factor of 4. This should fit on a 80x25 terminal nicely.
                        value[i] = value[i] / 4;


Position the cursor at the start of the bar
                        move(10 + i + i, 12);
For each position in the graph, either draw a * to show the value or a space to remove any * that might be there from a previous value
                        for(j = 0; j < 64; j++)
                        {
                                if(j < value[i])
                                {
                                        addch('*');
                                }
                                else
                                {
                                        addch(' ');
                                }
                        }
                }

                refresh();
 Check the keyboard and process the keypress
                key = getch();
                if(key == 43)
                {
                        command[1]++;
                }
                else if(key == 45)
                {
                        command[1]--;
                }
                else if(key > -1)
                {
                        break;
                }
        }

Shutdown ncurses
        endwin();
        close(fd);
        printf("%d\n", key);
        return(0);
}


To compile this program you need to use a new flag -l which says to link with the spcecified library (ncurses)

gcc -Wall -o pcf8591d-graph pcf8591d-graph.c -lncurses

When you run the program you should see something like this:
PCF8591


Brightness  *****************************************************

Temperature *******************************************************

?           *********************

Resistor    *****************************************

While it is running the graphs should move as you change the inputs. Try for example shining a torch on the LDR or adjusting the Pot.

You can adjust the green LED with + and - (hint, use - to go from 0 to 255 for maximum effect). Any other key will cause the program to quit.

The graph shows nicely how the inputs can change but it also shows how the value can fluctuate without any input changes. I can't explain exactly why but I would expect much of the fluctuation is because of the lack of a stable external clock/oscillator and/or instability in the reference voltage. Needless to say this is a low cost demo board and may not be exploiting the full potential of the PCF8591.

Here is the complete source code (pcf8591d-graph.c):
#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/ioctl.h>
#include <linux/i2c-dev.h>
#include <ncurses.h>

int main( int argc, char **argv )
{
        int i;
        int r;
        int fd;
        unsigned char command[2];
        unsigned char value[4];
        useconds_t delay = 2000;

        char *dev = "/dev/i2c-1";
        int addr = 0x48;

        int j;
        int key;

        initscr();
        noecho();
        cbreak();
        nodelay(stdscr, true);
        curs_set(0);
        printw("PCF8591");

        mvaddstr(10, 0, "Brightness");
        mvaddstr(12, 0, "Temperature");
        mvaddstr(14, 0, "?");
        mvaddstr(16, 0, "Resistor");
        refresh();
        fd = open(dev, O_RDWR );
        if(fd < 0)
        {
                perror("Opening i2c device node\n");
                return 1;
        }

        r = ioctl(fd, I2C_SLAVE, addr);
        if(r < 0)
        {
                perror("Selecting i2c device\n");
        }

        command[1] = 0;
        while(1)
        {
                for(i = 0; i < 4; i++)
                {
                        command[0] = 0x40 | ((i + 1) & 0x03); // output enable | read input i
                        r = write(fd, &command, 2);
                        usleep(delay);
                        // the read is always one step behind the selected input
                        r = read(fd, &value[i], 1);
                        if(r != 1)
                        {
                                perror("reading i2c device\n");
                        }
                        usleep(delay);

                        value[i] = value[i] / 4;
                        move(10 + i + i, 12);

                        for(j = 0; j < 64; j++)
                        {
                                if(j < value[i])
                                {
                                        addch('*');
                                }
                                else
                                {
                                        addch(' ');
                                }
                        }
                }
                refresh();

                key = getch();
                if(key == 43)
                {
                        command[1]++;
                }
                else if(key == 45)
                {
                        command[1]--;
                }
                else if(key > -1)
                {
                        break;
                }
        }

        endwin();
        close(fd);
        printf("%d\n", key);
        return(0);
}

Programming I2C

Although you can perform simple i2c reads and writes using the command line tools i2cget and i2cset, for a more integrated approach you can use a programming language to talk to the bus.

The are dozens of languages which make claims about ease of use and learning etc. and I am sure you can program i2c from them.

What I will demonstrate here is the simple way to do it from c. Although I don't aim to teach how to program in c, I will try and explain what the code is doing so you can follow along even if you are new to c.

This will use some basic i2c read and writes as described at  http://www.kernel.org/doc/Documentation/i2c/dev-interface
We will also need to perform some IO Control (ioctl) which are i2c specific.

First we need some code to get us started. The #include basicly make certain function calls and constants available to the rest of our program.

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/ioctl.h>
#include <linux/i2c-dev.h>

All of our code will live in the main function for now. main() is where all c programs start.

We need some variables which must be declared at the start of the function.
int main( int argc, char **argv )
{
        int i;
        int r;
        int fd;
        unsigned char command[2];
        unsigned char value[4];
        useconds_t delay = 2000;

        char *dev = "/dev/i2c-1";
        int addr = 0x48;

The [ ] syntax means an array so char command[2] is actually a variable which can hold 2 char values.

Some of our variables have been initialised to specific values so they are ready to use. The ones which have not been initialised will contain random values so we must assign a value to them before they can be used.

0x48 means hexadecimal 48 (which is decimal 72).

Next we print out a banner to show that the program is running
printf("PCF8591 Test\n");

The we get down to business and open the i2c device
        fd = open(dev, O_RDWR );
        if(fd < 0)
        {
                perror("Opening i2c device node\n");
                return 1;
        }

and select our slave device
        r = ioctl(fd, I2C_SLAVE, addr);
        if(r < 0)
        {
                perror("Selecting i2c device\n");
        }

Now we have an infinite loop
         while(1)
        {
There will be no way to end the program except by pressing Control C.

Next we have another loop which will run four times
                for(i = 0; i < 4; i++)
                {
Then we build a command for the pcf8591. The value of this is specified in the data sheet http://doc.chipfind.ru/pdf/philips/pca8591.pdf

In the first 8 bits of the command we will enable the analog output bit (0x40) and select which of the 4 inputs to read ((i + 1) & 0x03). We do a bitwise or to combine these values together with the | symbol.
command[0] = 0x40 | ((i + 1) & 0x03); // output enable | read input i

The // is the start of a comment so you can explain you code to the reader.

In the next 8 bits we increment the value for the analog output
command[1]++;

Now we are ready to send the command to the i2c bus
r = write(fd, &command, 2);

It is not clear why, but we need to wait for the command to be processed
usleep(delay);

Now we are ready to read a value. Remembering that the read is always one value behind the selected input (hence the +1 we used above).
r = read(fd, &value[i], 1);
if(r != 1)
{
        perror("reading i2c device\n");
}
usleep(delay);

Then we end the loop
        }
and now we can print out our results
        printf("0x%02x 0x%02x 0x%02x 0x%02x\n", value[0], value[1], value[2], value[3]);

end our infinite loop
    }

and although we may never reach here, we will clean up and quit.
  close(fd);
  return(0);
}

Now, if you enter all the code into a file called pcf8591d.c (you can copy the complete code as show below) then you are ready to compile it with this command
gcc -Wall -o pcf8591d pcf8591d.c
This says to compile the .c file and write the output (-o) to pcf8591d (if you don't specify an output file the default of a.out will be used which can be a but confusing). -Wall will make sure all warnings are printed out by the compiler.

Assuming the compile (and link) was successful you are ready to run
./pcf8591d
and you should see output like this:
PCF8591 Test
0x5f 0xd3 0xac 0x80
0xc1 0xd3 0xae 0x80
0xc1 0xd3 0xb0 0x80
0xc1 0xd3 0xb2 0x80
0xc1 0xd3 0xb7 0x80
0xc1 0xd3 0xba 0x80
0xc1 0xd3 0xde 0x80
0xc1 0xd3 0xdc 0x80
0xc1 0xd3 0xe0 0x80
To stop the program press ^c (Control + C).

Here is the complete source code:
#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/ioctl.h>
#include <linux/i2c-dev.h>


// Written by John Newbigin jnewbigin@chrysocome.net
// Based on info from http://www.kernel.org/doc/Documentation/i2c/dev-interface
// And http://doc.chipfind.ru/pdf/philips/pca8591.pdf

int main( int argc, char **argv )
{
        int i;
        int r;
        int fd;
        unsigned char command[2];
        unsigned char value[4];
        useconds_t delay = 2000;

        char *dev = "/dev/i2c-1";
        int addr = 0x48;

        printf("PCF8591 Test\n");

        fd = open(dev, O_RDWR );
        if(fd < 0)
        {
                perror("Opening i2c device node\n");
                return 1;
        }

        r = ioctl(fd, I2C_SLAVE, addr);
        if(r < 0)
        {
                perror("Selecting i2c device\n");
        }

        while(1)
        {
                for(i = 0; i < 4; i++)
                {
                        command[0] = 0x40 | ((i + 1) & 0x03); // output enable | read input i
                        command[1]++;
                        r = write(fd, &command, 2);
                        usleep(delay);
                        // the read is always one step behind the selected input
                        r = read(fd, &value[i], 1);
                        if(r != 1)
                        {
                                perror("reading i2c device\n");
                        }
                        usleep(delay);
                }
                printf("0x%02x 0x%02x 0x%02x 0x%02x\n", value[0], value[1], value[2], value[3]);
        }

        close(fd);
        return(0);
}
In the next blog I will improve the output to print a graph of the values so you can see them move up and down.

Adding Drupal nodes with a specific ID

As part of a drupal upgrade/migration I had a requirement to create a few thousand nodes with specific IDs. The internet said it could not be done but I did not believe it.

After diving into the bowels of the drupal code I found the spot which did the creating and it seemed to be doing something quite reasonable with the new node ID.

With just a small amount of convincing I managed to get it to do what I wanted.

diff -ru vanilla/drupal-7.12/modules/node/node.module web/drupal/modules/node/node.module
--- vanilla/drupal-7.12/modules/node/node.module        2012-02-02 09:03:14.000000000 +1100
+++ web/drupal/modules/node/node.module     2012-05-28 16:37:36.827171000 +1000
@@ -1095,6 +1095,12 @@
     if ($node->is_new) {
       // For new nodes, save new records for both the node itself and the node
       // revision.
+if(isset($node->request_nid))
+{
+       echo "Requested save new node with nid {$node->request_nid}\n";
+       $node->nid = $node->request_nid;
+       //print_r($node);
+}
       drupal_write_record('node', $node);
       _node_save_revision($node, $user->uid);
       $op = 'insert';

Creating a node can now be done like this:

$node = new stdClass();

$node->type = ...;

node_object_prepare($node);

// fill in node details here...
$node->request_nid = $my_nid;
node_save($node);

That seems quite trivial now. I don't know why so many people say it can't be done.

/usr/bin/ld: cannot find -lm

I was updating some of my software to run on CentOS6 and I had an unexpected error:

/usr/bin/ld: cannot find -lm
collect2: ld returned 1 exit status

The tool I was building uses -static and it turns out that static libraries are no longer shipped in glibc-devel. They are in a new package called glibc-static.

yum install glibc-static

solved the problem. My program now compiles and thanks to puppet, glibc-static is now installed on all my systems.
package { 'glibc-static': ensure => present }