Tuesday, 30 April 2013

Lexical Closures

Today, I found the first confusing bug in the software I've been working on. It had me going for pretty much the whole day, not helped by the fact that my MonoDevelop also had a bug causing it to crash when the IKVM runtime threw a meaningless exception, meaning I couldn't actually get to see any of what was happening.

I've been using a fairly out of date MonoDevelop, as it's the one available from the ubuntu repos. Strangely enough, recent builds on the website are only available for Windows, OSX and Suse.
Luckily, there doesn't seem to be any proper reason for this, so I basically did what this page told me to and built MD 3 from source. So far, it works a treat and doesn't seem to feature the annoying bug I mentioned. Nice :)

Anyway, the bug. It seems really stupid now, so in order to feel less stupid I'm going to turn this into a post where I explain lexical closures in the context of C#.

So, what is a lexical closure? Why, it's a function object which captures the bindings of free variables (variables which aren't defined in the function or it's parameter list).
I'll elaborate:

Delegates

If you've written C# before, you've probably used function objects or delegates, as they call them:

// Declare the delegate type outside a function somewhere:
delegate int OneAdder(int a);

// Instantiate the delegate
OneAdder addOne = delegate(int a) { return a + 1; };

// Use the delegate to print a "4"
Console.WriteLine(addOne(3));


You can also use the lambda syntax to more succinctly create delegates:

OneAdder addOne = (a) => a + 1;

There's more; From .NET 3.5 there are some standard generic delegate types already predeclared:

Action<int> : void-returning function with an int parameter
Action<int, int> : void-returning function with two int parameters
Func<int, string> : string-returning function with an int parameter
Func<int, int, string> : string-returning function with two int parameters

As you might intuit, delegates are reference types just like classes and amenable to the same assignment/passing-around semantics. For instance, one could write a function:

static int CallMyDelegate(Func<int, int> func)
{
 return func(5);
}

Lexical Closure

Delegates in C# have 'Lexical closure'. That means that when you call this, passing it 5:

static Func<int, int> MakeAddingFunction(int number)
{
 return delegate(int a) { return a + number; };
}


What you get back is a function which adds 5 to things.
It's called a lexical closure, because the function 'closes over' the 'number' variable (which is in its lexical scope), so it can use it later. One can also say the delegate 'captures the lexical environment'.
Think on that for a while until you understand it. Try it out, even.

This whole delegate thing is really useful as it gives us a nice, succinct way to abstract some operation in terms of it's inputs and outputs, so it can be passed to code which can then apply it to some data without understanding the operation. C#'s "Linq" is based on this kind of thing. If you haven't heard of Linq go and read up on it once you've finished reading this.

Peculiarities

Lexical closure is about capturing the *bindings* of variables in scope, not the *values* of them.
The binding is the invisible thing which is given a name and associated with different values over the variable's lifetime.
This means that the delegate has full access to the variable, rather than just receiving some copy of its value.
Consider this code:

static void Main(string[] args)
{
 int someNumber = 3;
 Action<int> addThis = delegate(int a) { someNumber = someNumber + a; };
  
 addThis(10);
 
 Console.WriteLine(someNumber); // This prints "13"
}


That's right - it prints 13! The someNumber variable was modified by addThis.
Whenever you open up a function, a loop, or an if block, or anything with curly braces, that conceptually creates a new 'lexical environment' and a new set of bindings for all variables declared within those braces. In the case of loops, a new environment is created for each iteration, so you can bind to different variables each time round, even though they have the same name:

var setters = new List<Action<int>>();
var getters = new List<Func<int>>();

for(int i = 0; i < 3; i++)
{
 int closedOverVariable = 0;
 setters.Add(delegate(int newValue) { closedOverVariable = newValue; });
 getters.Add(() => closedOverVariable);
}

// Set the first int to 5
setters[0](5);

// Set the second int to 100
setters[1](100);

// Print the value of the first int : "5"
Console.WriteLine(getters[0]);

// Print the value of the second int : "100"
Console.WriteLine(getters[1]);

We just created three ints with no way to access them apart from via the appropriate getter and setter delegates. I always found that kinda spooky; It's like the ghost of a variable...

The Bug

So where did it all go wrong today? Here's a piece of code which demonstrates what I got wrong:

// A nice array of ints
int[] ints = new[] { 1, 2, 3 };

// A list of delegates
var delegates = new List<Func<int>>();

// Lets go through the ints, and make a 
// delegate for each one which returns the corresponding int.
foreach(int i in ints)
{
 delegates.Add (() => i);
}

// Now go through all three delegates, printing the return values.
// It should go "1, 2, 3", right?
Console.WriteLine (delegates[0]());
Console.WriteLine (delegates[1]());
Console.WriteLine (delegates[2]());


WRONG! It will go "3, 3, 3"!

That's because the foreach statement in C# doesn't create a new binding for 'i' at each iteration. It behaves more like 'i' is declared outside the loop and the value reassigned 3 times. Each new delegate we're creating is getting the same binding - accessing the same variable. I find this a little counter-intuitive, as I would expect there to be 3 different bindings created in 3 different lexical environments as we iterate over the ints, just as there would if 'i' had been declared inside the loop.
That last part is actually the way to fix this mess:

int[] ints = new[] { 1, 2, 3 };

var delegates = new List<Func<int>>();

foreach(int i in ints)
{
 // 3 different bindings will be created for 'tmp'
 // in 3 different lexical environments.
 // We're initializing it with the *value* of 'i'
 int tmp = i;

 // Close over the binding of 'tmp', rather than that of 'i'
 delegates.Add (() => tmp);
}

Console.WriteLine (delegates[0]());
Console.WriteLine (delegates[1]());
Console.WriteLine (delegates[2]());



And that's all, folks. Hopefully I've helped someone understand lexical closures better. Maybe I've even managed to steer someone away from falling into this trap.

Happy coding :)

Monday, 29 April 2013

More TestVisor Development

Firstly...

Git

I'd like to point out that Git is pretty darn good.
I picked GitHub for the hosting because I liked the look of it and had heard ravings about Git; This is probably one of the best decisions I made for this project.
I've only had extended contact with Mercurial and Subversion in the past and I can definitely appreciate the difference in target markets:
Git is targeted toward people who value a holistic view of their environment; It's a more toolbox than black box. Mercurial(or Hg), is more of a nice extensible black box with a good set of buttons on the top.
Any software dev team should really be familiar with at least one of these tools.

IKVM

IKVM seems like a magical piece of software. It's basically a JVM running on CLI/.NET, and it works!
I had a problem - I'm developing in mono, but I was lusting after Mozilla Rhino, a nice Javascript engine written in Java. After a quick look around, I didn't really find anything that worked as well for my platform, so I went ahead and downloaded Rhino. One call to "ikvmc", and my problem seems well and truly solved! I got a Rhino.dll that I can link into my mono project with the worst of the hassle being that I had to also include /usr/lib/ikvm/IKVM.OpenJDK.Core.dll (to use of some jvm types like java.lang.Class), which took around a minute to locate and reference.
I've gotta say, it felt weird typing this into a C# project:
return Context.javaToJS(new java.lang.Boolean(func(testParams, testKey)), s1);

TestVisor

It's been an exciting few days for this project. I'm still on holiday, so I've been tinkering with it pretty much nonstop.
I'm hopefully getting some help with the work from a University chum before long.
Here's a little demo of Javascript test plans:



Tuesday, 23 April 2013

TestVisor passes its first test

TestVisor

I've been working away on this little project for a while as an exercise in coding something useful in my free time.
The end result will be an ajaxilicious virtual machine hypervisor manager which will be able to run sequences of tests on a virtual machine instance,
making use of snapshotting to deal with real world automated testing in a state-controlled way.

It's a long way off done, but here's a taster:



Here we see it initializing some stuff, powering up a Windows 7 Virtualbox instance, downloading a test, executing the test and uploading the results.

My holiday started today and I've been hacking away on this since the morning, I'm quite pleased with what I've got so far. I've been poking this on and off for a few weeks now, but there's truly no substitute for 'the zone' which I finally managed to get into with this.

I was also pleased to find that I could compile a mono executable on the host linux machine I'm developing on and download it to the Windows 7 VM to run under CLR with 0 problems.

It's on GitHub, but not in an amazing state currently. Largely stubs, test code, commented out test code and some hints of where it's going.

Exciting to finally get another project underway, I can't even remember what my last one was...

Saturday, 20 April 2013

Being an OOP Expert

It's tough to be an expert at Object-oriented programming. There all kinds of things you have to know about, like singletons, visitors, abstract superclasses, methods, factories, facades, the bridge pattern...
If you don't know what these things are, how can you ever hope to produce scalable, maintainable software?

A Job Interview

At the interview for my current job, I was presented with a matrix to fill out. The rows were different skills or technologies, and the columns went from "Don't know about" to "Expert" or something similar.
Amongst others, there was a row for Unit testing, a row for .NET Framework and a row for Object Oriented Programming.
I ticked the "Expert" box for this row only.
Now you're probably thinking how arrogant I am. Some of you, maybe those versed in CLOSMultimethods or Haskell's typeclasses are probably already thinking of how best to blow my mind with a humbling comment.
That's why I'm explaining now, that I'm aware of the number of different technologies and practices which "Object Oriented Programming" can be made to refer to. I'm also aware that those technologies are almost certainly not the ones they mean when they're asking how much I know about OOP.
For the rest of this post when I'm speaking about OOP, I'm referring to what 90% of the software world know as OOP; That is a class-based, stateful object system with information hiding annotations and subclassing. Java, C# or C++ are the systems I speak of.


I'm not convinced

At the risk of sounding arrogant all the same, I'd say I've written plenty of code in Java and a reasonable amount of C#, I've been to hell and back with this idea of OOP.
I used to be an OOP proponent. A zealot.

Once, I accidentally used a 2D array to represent a matrix for a scholarship project. I was advised by my supervisor to do something more "object oriented" such as represent a row with a class and use a Java List<T> for the columns.
This greatly improved the scalability and modularity of my code, so much so that after the project, my code was bought from me for a large sum of money by a multinational firm, and it currently sits at the core of a complex probabilistic model used to drive the world's cruise missiles. Couldn't have done that with a mere 2D array, could I now?
Ok, I promise I'll stop the sarcasm now.

"More object oriented"...

What I was really being advised to do was to make use of a larger number of the OOP constructs provided by the language. I went ahead and did this, as it seemed like a good idea at the time.
Like any nerd I like shiny things and by god, the result was that.
It was also completely nonsensical to anyone who actually understood how the solution worked.
This is what happens when the solution you want to implement gets really departed from the terms you're implementing it in.
This is actually the idea behind Domain Specific Languages (DSLs), where the language is designed around concepts involved in solving the problem.

I had an excuse - I was 16 years old and I had just started really playing with OOP, falling into all the traps, spending most of my time figuring out how to represent my solution with sets of classes, and generally going nuts with the whole idea.
This is not a professional or productive way to work. This kind of thinking is not engineering.

Programming

To me, programming is about finding the route from the problem to the logic gates in the processor. It's a game of representations, that is, abstractions. Abstractions are necessary so you can make large problems manageable by putting them into simple terms. As a software engineer, your job is often to design these abstractions and make them useful in as many situations as possible, whether that's by extending a language like Java with a fancy library, or creating a new DSL over a host syntax like XML or even with its own custom parser.
This is a big part of what makes software engineering so hard; There are many ways to look at a particular problem.

Putting aside the awkwardness we know about, you might think representing that matrix as a list of column objects would be a reasonable idea. But just wait until someone wants to build a system on top of it which needs to look at matrices row by row or god forbid, even diagonals.
It's not intractable; You can indeed wrap it up in a "float[] GetMatrixRow(int rowIndex)" function, but that's another layer of abstraction to maintain, another layer between what the compiler reads and your actual intent(bye bye code optimizations), and more code to drown in when you can't figure out where that NaN is coming from.

Another problem with building a code tower like this is now that your actual "intent", whatever that was, is now just some part of this mass of abstraction. As far as the compiler is concerned, it's a lump of code. It'll compile it, but there's not going to be any reasoning about it.
What you're writing is not something which describes your solution particularly well, rather something which after being run will produce correct behaviour. This makes the implementation less valuable, in my opinion.
This all sounds a little extremist I'm sure, but keep in mind the problem I posed out was an extremely simple one - doing something columnish with a matrix of numbers.

erm... OOP?

Yeah yeah, I know, I've been banging on about domain specific languages and their benefits for a good number of paragraphs now; This was supposed to be about OOP.
OOP represents an extension of what gets commonly called the "imperative" programming model;
The "Do this, now do this to change that, now check if that's changed, do these things if it has" model, which should really be considered a pile of fairly sophisticated syntactic sugar for machine code, as it really isn't all that far departed from the basic Von-Neumann model which is what's actually happening in your computer case.

OOP is an extra layer of sugar up, and a slight move into another dimension - there is an introduction of some extra concepts which aren't just about sequenced operations.

Here's the thing: In order to solve most any real world problems, you have to build abstractions atop your processor. Unless your problem is to add 3 numbers in sequence together for which, you could say your processor supports hardware acceleration :P.

Some problems are solved by computing linear functions to calculate what direction your cruise missile is travelling in.
Some are solved by running an optimal sequence of dependent steps in the most efficient way, such as a build system (Makefiles are a good example of this).
Some need to keep track of groups of values which evolve and affect one another in predefined ways, such as a particle system.
Wikipedia tells me that OOP came about trying to solve problems similar to the last one.

It's no secret that as long as your machine abstraction(model) is Turing Complete (TC), you can represent any computable function. That goes for OOP too; There's no saying it's impossible to code certain things in an OOP or non-OOP TC language.

The point I'm making is that having the OOP abstraction as part of your language really isn't that applicable and useful by itself, and it's definitely not something which will increase productivity and scalability for most classes of problems.
Unless your problem is to represent the state of synchronous message-receiving groups of values, it won't solve much of your problem either. Yes, I know you can use those constructs to represent a solution to the problem, but you could also probably build a lot of furniture with hammers and nails. If you've got a hammer available, that's great, but it doesn't mean it's time to start either building nail-oriented furniture, or building more appropriate tools by nailing multiple hammers together to create some kind of saw and......
You get my drift; Go and buy more tools. :)

Patterns

So, you've got an OOP language: A language with first-class support for managing groups of values and synchronous messaging between those groups of state.
Great, Now you want to solve your problem with it.
Design patterns to the rescue!
Yes, conceptual adaptors if you will, to convert the model your OOP language gives you into something you can use to solve a common class of problem. There are some patterns which are only a little extra atop the OOP model. Other patterns however, are quite a jump. The Visitor pattern, for instance. Now I'm not going to start whining about the Visitor pattern and how complicated and unreadable it is to implement, because that's not important. What's important is how readable the result is, and in my humble opinion, the result works quite nicely once you look past the boilerplate. This is important to my point, as I can now point out that having that OOP model somewhere in there didn't really help us at all. It didn't bring us any closer to traversing that tree or whatever we used the Visitor pattern for, if anything it got in the way a bit.

It got in the way a bit

That's what I commonly find with OOP ways of thinking. I've seen some absolutely ghastly code which was written in a way which seemed to say: "Look, I managed to find a use for OOP here, here and here!", in fact I've written a fair amount of it. I'm not saying that OOP is the devil, because it is not. It's simply a model for solving certain classes of problems.
So if OOP isn't going to be the centrepiece what concepts should a language designer make first-class citizens in a "General Purpose" Programming Language?
You can go for pure functional, which can be a pretty close fit for some problems, but of questionable use for matters of state.
There's the C#/C++ approach of including as much as possible. Clojure is a really good example of this, but the rules are a bit different there as it's a Lisp, and the distinctions between language features and library features tend to disappear.

Wait!

I haven't mentioned a lot to do with OOP, I know. I haven't really talked about encapsulation that much. I haven't talked about subclassing or polymorphism.
How on earth can this be a post about OOP without those things?
My answer to that is that it's not really a post about OOP. It's a post about representing problems, the misunderstood role of OOP as the saviour of software complexity, and why I think I can write that I'm an "Expert" on OOP.
I can write that I'm an "Expert" on OOP, because I know what it is and what it's not. I know when it's appropriate, and I know when it just gets in the way a bit. I also know that it certainly doesn't get the engineer out of having to solve the actual problem!


This turned out longer than I expected, I'm now off to watch today's Doctor Who ep.
'Til next time.

Back again

Yesterday was friday. I got down on friday a little too much by sleeping for about 14 hours after work. Here I am on Saturday, and I'm sitting in a Starbucks committing a new post to my long-untouched blog.
I feel like I've got a couple of things to write about, so I should do it properly - with smooth jazz and a large latte from the oddly friendly baristas.

Gi'z a Job?

When I wrote my last post, I was about to embark on a grand tour of england to ask ~10 companies for work. That was around 2 years ago, and turned out quite well. The first interview went quite well, I did their test, presented myself professionally(or so I thought), but apparently they didn't like me.
As it happens, I only recently figured out how I managed to make a pigs ear of that one. It was like a lover's quarrel - one unnoticed technical misunderstanding lead to a nonsensical quip and before long, the interview had spiralled out of control while I, obliviously, was talking out of my arse with such prowess I'm surprised they didn't hire me for a sales position (hurr hurr).
I visited a number of other companies including Altera(great place to work by the looks of it. Would not have been a good fit for me at the time). I ended up at a great little firm in Bristol: Multicom. Not many people, lots of work; I couldn't have asked for a better first job.

'Professional'

I like to think I became a professional in the six months of employment which followed. I learned the ropes, I learned the shape of things to come, I learned how my skills fit into a business and what it's like to blend fun with that which makes money.
Being honest, I went into there almost suspecting that I was going to be nothing more than a drain on resources and general annoyance. Quite the opposite happened - I gained a huge amount of confidence and it turns out that employment was what I needed to start enjoying life.
After six months, pounding out Java to mine data from all kinds of sources from clean XML APIs, ghastly XML APIs, nonsensical websites and excel spreadsheets(http://en.wikipedia.org/wiki/Apache_POI is decent for this),
I moved back to Aberdeen for unavoidable family-type reasons(I wasn't actually homesick).

Oops, I learned something

It turns out it was much easier to get a job the second time round. With my CV bulked up with all kinds of experience, I was quickly snapped up and now spend most of my time working on fancy geology software.
I've worked there for about 16 months, now.
I have no regrets regarding my decision to take work instead of another year at university and an honours degree. It suited me, it might not suit you, or someone at another university on a different course.

Things I noticed along the way:

  • Employers are scared of graduates (rightly so, given what a joke some modern CS courses are).
  • It's good to customize your CV for different kinds of work you're willing to do. I personally kept 2 CVs around. One, I like to call my J2EE CV, which boasts of Enterprisey experience and my favourite Java features. My other CV has more of a tinkerer feel to it, talking about how I'm used to a C/UNIX environment and like to play with microcontrollers.
  • Software Engineering is a young profession and as such, there are lots of cults, misinformation, obsessions and unknowns. It's also strangely hard at times.
  • It's good to stuff your CV with buzzwords, so long as they're accurate: Your CV will almost certainly pass under the nose of someone who is looking for them.