Half-technical discussion on using Abstraction, APIs and a bit on Lisp

Why abstract?

Abstraction is about generalisation. it allows us to do more, while knowing less. If a baby is given some beans to hold, it may not be long before they discovered that transferring a bean from one hand to another means the number of beans in the from-hand goes down, and the number of beans in the to-hand goes up. They may also figure out the process is reversible, and transferring a bean in the opposite direction leaves them equivalent to where they started in terms of per-hand bean-quantities. There's a common abstraction to deal with such situations, known as arithmetic. Arithmetic is very much an abstraction, because most of the time it doesn't deal directly with what you're using it for. Arithmetic works in terms of numbers. Counting or estimation can map these intangible entities onto something real. As long as we know how to count the various different real life things we might use, everything we know in terms of 'arithmetic' is useful, and can help us solve our problem.

Abstraction is an important part of software, as with any complex system built with or used by limited human brains. Actually creating these abstractions, I would have no qualms calling an art form. What's different about this art form however, is that the product has to be actually useful to someone.

Programmers make use of abstractions all the time. A common example would be structured programming, that is, programming with 'functions'.

In most cases, a function is a sequence of operations with an associated lexical context. By lexical context, I refer to the ability to give things names and refer to those things using the same name, but only while inside the particular function.

Functions and lexical contexts aren't tangible, yet a structured programming language is perfectly capable of describing these things. They are an abstraction, and a useful one, born out of a common need to break problems up into simpler parts. Before functions were formalized in structured programming languages, this behaviour would have been implemented using a lower level abstraction, such as machine code to manage groups of related values, which is itself an abstraction built atop electronics in the computer's processor.

Most computer programs are written in terms of functions. Your web browser would likely have hundreds of thousands of functions; There will almost certainly be one to calculate the width of each character on this page, for example.

Stacking abstractions

Abstractions are often layered; That is to say, it may be useful to define one abstraction in terms of another. Natural language is as good example as any;

English-speakers make various sounds, yet the English language doesn't apply meanings directly to these sounds, but instead defines 'words' in terms of these sounds. A major advantage to this is that it makes the English language 'portable'; You can choose to either speak the word "house" audibly, or write it down, and it still has the same meaning.

Grammar is above still, defined in terms of word types(adjectives, verbs...). These word types are in themselves defined by the words included in them.

Building up abstractions and dissecting them really isn't an exact science; There really is more than one way to skin a cat.

Programmers often build abstractions atop these functions to represent higher-level concepts, such as this text. If creating a web browser, they would likely want to create an abstraction to represent visible things. This would be useful, as there are all kinds of different visible things on a web page, which are also used in similar ways; Being drawn on the screen, for example, is something which happens to all visible objects. Being clicked doesn't make sense for all visible objects, meaning that deciding what to do when a mouse clicks the item may not have a place in this particular abstraction.

This abstraction could be built in terms of C# functions by, for example, creating a function named "Draw", for each different kind of visible object, whose job it was to draw the particular thing. This way, other parts of the program wouldn't need to deal with how many curves and lines make up a "Subscribe" button, they would only need to be able to run the appropriate "Draw" function for the visible object.

This is a very simple example, and you would likely want to do more than just draw a visible object. 

You would probably also want to be able to calculate the shape and size of a visible thing's outline before calling its "Draw" function. With this information, you could decide where it should be drawn so it doesn't collide with other drawn things.

We're separating the "How" from the "What". We're allowing one to draw a button or other visible object("What"), without needing to know "How" to draw it. In fact, given the right machinery, it needn't know it's a button at all. Just like we can reason about bean quantities without learning how beans actually work, because counting and estimation operations are really all that's required for the "arithmetic" abstraction.

APIs

API stands for "Application Programming Interface". An API is an abstraction built on top of a programming system, often in terms of functions, used to achieve some end. You could argue that our "visible object" abstraction above was a kind of API and I think you would be right. An interface in the broadest sense is a view through which you can interact with some system or another. Dead simple real life example is a headphone socket (Headphones don't care if they're playing rock or jazz, because that's not the abstraction they are designed to).

Modern programming languages offer many abstractions besides just functions, including:

• Interface - A set of functions which operate on the same item of data.
• Array - A sequence of data of a certain kind.

Tower of abstraction

You may have noticed the trend to build new abstractions in terms of old ones.
Interfaces, for example, are based on functions; To use an interface, you need to know what an interface is, and also what a function is, and all about functions which, in some programming languages like C#, can be quite complex creatures.

Looking at an abstraction like the C# programming language, it's mostly/completely isolated from what's below. It has its own data types, terms and semantics that you can build solutions from.
I think here, the 'purity' of certain abstractions becomes most apparent. 

Unfortunately for the programmers, the domains of the problems they're *actually* going to be solving often have nothing to do with constructs such as strings, classes or network sockets, meaning that you still have a lot of work to do in terms of supporting the solution somewhere above the programming language by building further layers of abstraction.

Visual Basic was a programming system developed by Microsoft which, I would say, focused on bringing the level of abstraction closer to the problem domain than many systems. A major example was the notion of the "Form" - a modular interactive graphical container to hold "controls"(Buttons, text boxes...). These things were represented in the Win32 API, but the drag-and-drop form designer allowed you to create GUIs not by way of calling "functions" or talking to "objects" via "interfaces", but rather by...well...creating a GUI, in the most direct way imaginable.

I'm not saying that Visual Basic was the best all-round programming system I've ever solved a problem with(far from it), but these kind of things made it very useful indeed.

HTML is another common example of this kind thing. You don't call functions and assign event handlers to create DOM nodes representing visible elements, rather you use HTML to directly create DOM nodes which represent visible elements. The reason there's still another layer there, and no good(tell me if you know one) drag-and-drop designers for the web is that the web isn't about positioned components on a page, it's about a flexible, reflowable, semantically rich layout which doesn't care too much about screen size. It's not quite as perfect as that last sentence makes it sound, but I do believe we're getting there... :).

These very "pure" abstractions which stand on their own, making use of only constructs which are part of a problem domain are often called  Domain Specific Languages(DSL) .

Domain Specific Languages

There is really no clear definition of what constitutes a DSL; it's very much down to intent, I believe.

Some examples of successful domain specific languages:

• Makefiles
• HTML
• Most things built on XML

I'll point out that you can still effectively hide intermediate levels of abstraction without writing your own parsers or using XML; You can indeed embed clean abstractions in programming languages using relatively simple concepts like arrays and meaningful parameter names in function calls as a backbone, and build a language like that.

Waste of time

There are disadvantages to this approach; Even if your discipline and methodology and separation-of-concerns is perfect, jumping through all of those abstraction layers(which you'll have to do at run-time), will have a performance impact, a major one sometimes; Look at the speed of Win32 UIs vs Java Swing UIs - Swing loses...badly.
What you can end up with is basically a poorly performing interpreter.

Sure, there are optimizations you can do to mitigate this, like runtime code generation using things like Javassist, LLVM or .NET's IL-generation classes, but a far simpler approach which doesn't leave the user waiting for their program to compile the rest of itself(which is essentially what will be happening), is to take care of this abstraction level at the appropriate time - The time when you should be doing all of the mindless work to connect what you've written to something which isn't going to need to change at run-time, the time used to convert abstractions into their run-time form: Compile time.

This is why C has macros, why C++ has templates, why Javascript has eval(), and one of the best things about Lisp.

Lisp for DSLs

Lisp (and Smalltalk so I read) implementations make this kind of thing very easy, as they don't generally distinguish between compile-time and run-time; Your build process consists of running a bunch of code to set up the state(such as code to define functions), then you image the whole runtime, serializing all the state, and ship it like that.

This means that if you did wish to implement a DSL with data structures, you could process those structures as part of your build, making it far closer to 'source code', which is what it is.

Lisp goes a step further than smalltalk and ONLY uses data structures for source code, making it very easy to implement your own languages using the available syntactic constructs with your own semantics.

Lisp object notations are normally very lightweight;

Commonly, list data is represented with something like: (1 2 3 4), a function call taking the form: (functionname arg1 arg2).

Clojure, a fairly modern Lisp which is gaining popularity, has a rich set of first-class data structures including lists: (1 2 3), vectors: [1 2 3], sets: #{1 2 3} and maps: {1 "a" 2 "b"}.

These things come together to make a great environment for DSL construction.


That's all I've got to say today. This post has been sitting unfinished in drydock for a while, I like to think it's more understandable as a result of the extra thought.