Noel Rappin Writes Here

So, after writing two articles basically assuming what Ruby’s influence are, it occurred to me to check the About Ruby page on the official Ruby site.

It says this:

Ruby is a language of careful balance. Its creator, Yukihiro “Matz” Matsumoto, blended parts of his favorite languages (Perl, Smalltalk, Eiffel, Ada, and Lisp) to form a new language that balanced functional programming with imperative programming.

We’ve already talked about Perl and Smalltalk. I don’t know much about Eiffel or Ada, though I assume Eiffel inspired some of Ruby’s object structure somehow. Ada is perhaps the most statically typed language in existence and it’s hard to see how it could have possibly influenced Ruby in any significant way, but I guess it must have.

Which brings us to Lisp. I’ve been a little nervous about approaching Lisp because, while I have actually done projects in Lisp, it’s been a while. And I assume there’s a whole cadre of Lisp-knowers waiting to jump on misstatements. Hi, Lisp-knowers!

I don’t even really know where to start… Let’s start with Lisp’s most prominent syntactic feature… If you know anything about Lisp, it’s probably all the parentheses.

Lisp has a very basic syntax, or at least the core of Lisp does. There’s no syntactic distinction between code and data.

Lisp code and Lisp data are made up of lists. A list is surrounded by parentheses and contains multiple atoms, separated by a space. Lists can be nested.

(1 2 3 4)

(1 (2 (3 4)))

(Once upon a time, I was told that you could always tell Lisp programmers during the push card days, because they’d have shirt pockets filled with right-parenthesis cards. Is this true? Probably not, but I love the story.)

Internally, the list is stored as a set of connected pairs. For the top list, the first pair is made of the atom 1 and a link to the next pair, which is made up of the atom 2 and a link to the next pair, and so on. Many languages have this structure under the name linked list.

For historical reasons, the first element of the list was often called the car and the remainder of the list is called the cdr, which is pronounced something like “could-er”. (More modern Lisps call these first and rest).

Okay, fine – car stands for something something “contents”, “address” and “register” and cdr similarly has to do with “contents”, “decrement” “register”, which are terms that would mean something if you programmed in assembly on an IBM 704 in, like, 1959. (Wikipedia just told me they aren’t simple acronyms, which is news to me and every other source I’ve learned Lisp from in the last thirty years.)

There are at least three strands of Lisp that are even a little viable today:

• Common Lisp, which is a direct descendent of the original Lisp from the 60s, and came out of standardization attempts in the 1980s. Common Lisp has mostly a basic syntax, but a very large number of standard functions and special cases.

• Scheme, which was created in the 1980s as an alternative to Common Lisp, with simpler internals and fewer special cases. Scheme is notable as the programming language in the original version of the book Structure and Interpretation of Computer Programs.

• Clojure, which is a Java VM based Lisp that originally released in 2007. It adds some extra syntax for different kinds of literal data. In particular, Clojure introduces an array-like list rather than a linked list-like list, this improves performance at the cost of making the syntax a little more complex.

All these are slightly different, and in particular they have slightly different syntaxes for very basic things, but we’ll muddle through.

Anyway, lists. In Lisp, lists are both data and code. In order to convert code to data you have to evaluate it. Lisp was (I think) the first language to be interactive with a REPL and you can get Lisp to evaluate your code in that REPL – this example is Clojure, which was the least-sketchy install on a modern Mac, but you can also use https://try.scheme.org to play around with Scheme:

user=> (+ 1 2)
3

user=> (+ 1 2 3)
6

user=> (+ (* 3 4) (+ 3 4))
19

user=> (first '(1 2 3 4))
1

In general, what happens here is that, when treated as code, the first element of a list is expected to be something function-like that can be applied to the rest of the elements of the list, so + adds the elements of the list. Nested lists are evaluated first.

That last example is doing something subtle with that quote character. Quoting was, I am now remembering, the bane of my existence when I was actually using Lisp. The quote character is a shortcut for an actual function named quote and what it means is “treat me as data, not as code”. So, in that last example, the function first is getting the list as a whole as an argument. Without the quote, the system would attempt to evaluate (1 2 3 4) and error because 1 isn’t function-like. (Not for nothing, but the clojure REPL error message in that case is really unhelpful).

What’s happening here internally is that the system is calling a function named eval with the list as an argument. The eval function looks at the first element of its argument and determines what to do based on that – most commonly the first element is the name of a function, in which case the function is executed using a method called apply.

Where this gets subtle is that these two expressions have the same result:

(eval (first '(1 2 3 4)))

(eval '(first '(1 2 3 4)))

But they are taking different paths. The top expression starts by evaluating the inner list with first and so it boils down to (eval 1).

The bottom expression actually takes the entire list as data, so it actually calls eval on (first '(1 2 3 4)) which also boils down to 1.

If I had fully understood this subtlety in, like 1990, my life would have been somewhat easier.

Instead of a function, the first element of a list could be something typically called a “special form”. A special form does something weird, it has atypical evaluation or a side effect or both. For example, all these Lisps have a special form for assignment and a special form for if. Here’s Clojure:

(def x 3)

(if (< x 5) 
  (format "%s is less than 5", x) 
  (format "%s is not less than 5", x))  

So, def is a special form because it has a side effect, it’s creating a local value that can be used later on, so it’s not evaluating x. And if is a special form because the format is if condition true false and it only will evaluate whichever one of those corresponds to the condition, the other branch will not be evaluated.

It’s my recollection of normal Lisp style that end parens are placed on the same line, rather than one to a line as we might do in Ruby. I also note that Lisp and Smalltalk are both languages where it was very common to work in non-monospaced fonts.

History and Lisp

Lisp is one of the oldest continually used programming languages going. Development on it started in the late 1950s, though in the beginning the parenthesized form was only one of the ways you could define expressions, that form quickly became the preferred, and then the only way.

I’m actually a little vague on what the Lisp world was like through the 60s and 70s. My programming book from 1975 mentions Lisp a couple of times in passing, but I don’t get the sense it was heavily used outside of academia during that time. Douglas Hofstadter mentions Lisp in Goëdel, Escher, Bach, published in 1979, and wrote some columns about it in 1983 for Scientific American.

Within academia, Lisp became the primary language for Artificial Intelligence research, largely on the theory that there was something profound about blurring the line between data and code. The idea was that you could create a program that could manipulate itself. Lisp remained the primary language of AI at least through the mid 1990s and probably further than that.

Lisp also became very closely identified with the early Free Software stuff, which lives on in Emacs Lisp being the configuration language of Emacs. And also in the way in which Common Lisp and Scheme are still distributed.

I also get the sense that in the 70s everybody and their postdocs had their own dialect of Lisp, and there was some not-fruitful chaos there. In the early 80s, work began on a common standard, and in 1984, Common Lisp the Language by Guy Steele was published, and that eventually became the basis for an ANSI standard Lisp.

Also in the 80s you had Lisp Machines, full-on hardware that used Lisp as their main language the way that Unix used C. It was possible to have Lisp implementation that were as fast as C implementations (or so I have read), though I suspect the Lisp hardware was more expensive.

(Here’s trivia for you – the company that made those machines was called Symbolics, and in March 1985, they became the first company to have an internet .com domain.)

I actually touched one of those Symbolics machines, there were still one or two of them hiding in the AI lab at Georgia Tech when I started, but they were pretty old, and I never did anything substantive on them.

Anyway, Lisp sort of slowly became less relevant to AI research and I don’t think it ever had much commercial pull. There was a really good Mac implementation in the System 7 era that was both quite fast and tied into the Mac toolkit in a way that made it possible to build real Mac-native software. (It came with its own editor called FRED, which stood for “Fred Resembles EMACS Deliberately”, because of course it did.)

Paul Graham of Y Combinator fame was a Lisp guy in his past, and around 2001 or so was trying to push Lisp as a killer language for startups. Not many people took him up on that.

Eventually, Clojure was released, and that did get some attention, and still survives to this day.

Me and Lisp

My time with Lisp was brief but meaningful. I first encountered a Lisp in the form of Scheme in my CS1 class, which used the famous Structure and Interpretation of Computer Programs as its text book. I was a kid who had mostly used Turbo Pascal. To say I was unprepared was an understatement.

It took me quite a while to get my head around Scheme and around functional programming generally.

A few years later, as a grad student, I spent some time working with Lisp in AI classes. (I was a TA for an undergrad AI class where, due to some schedule weirdness a significant number of people had not taken a Lisp intro course and we had to run an emergency catch up session). I also built a pretty decent sized project in Mac Common Lisp, which was both fun to work in and maybe the worst codebase I ever wrote. (I still didn’t understand objects, and the Common Lisp Object Model was not helping).

You and Lisp

I’ve been trying in this series to give you a sense of what it feels like to work in these languages.

One key thing about Lisp that I didn’t mention up above is that a key part of working in complex Lisp projects is the macro. To wildly oversimplify something that I barely understood when I was using Lisp regularly, a Lisp macro takes advantage of the permeable border between Lisp code and Lisp data. A macro is a piece of code that takes data which is not quite executable code and transforms it into differently shaped data which is executable code. By which I mean the macro takes a list, does something to it and literally returns a new list that is meant to be executable.

Why would you write a macro when you could write a function? Typically, it’s because you want to do write something like your own special form – something that breaks Lisp’s normal execution plan. Macros are also typically evaluated at compile/interpret time not at runtime, so they might be faster than functions.

I’m trying to find a decent example – not really being skilled enough to generate one on my own, and here’s one that might resonate with Ruby devs. From here – I tried to adapt to Clojure but failed.

Let’s say you were writing a lot of similar log functions – you could write them all as separate functions

(defun log-debug (message) (format t "DEBUG: ~a~%" message)) 

(defun log-info (message) (format t "INFO: ~a~%" message)) 

(defun log-warning (message) (format t "WARNING: ~a~%" message)) 

(defun log-error (message) (format t "ERROR: ~a~%" message))

That’s fine but you could also do this more metaprogrammatically with a macro:

(defmacro generate-log-function (level) 
  `(defun ,(intern (concatenate 'string "log-" (symbol-name level))) (message) 
  (format t "~a: ~a~%" ,(string-upcase (symbol-name level)) message))) 

(generate-log-function debug) 
(generate-log-function info) 
(generate-log-function warning) 
(generate-log-function error)

Without going into the syntax, the basic idea here is that defmacro takes in a level argument and generates a list that starts with defun and defines the same function listed above.

And so, a lot of writing complex programs in Lisp starts out by using macros and functions to turn Lisp into the language that you actually wanted to write the code in. And I guess that’s sort of true of most languages, but it’s more true of Lisp. The fact that Lisp programs are manageable as data makes incredibly easy to create your own language constructs. It also means that sufficiently complex Lisp programs are basically nothing like each other, aside from all the parentheses.

If Perl is like being handed a Swiss army knife with no handle, and Smalltalk is like being handed a whole computer, Lisp is kind of like being handed a bunch of very well designed, but small, Lego bricks. The good news is you can piece them together into anything, and the bad news is that you have to piece them together into everything.

As to what happened to Lisp, well, I’m not sure anything in particular happened. It was basically an academic language that also was big in the early open-source community. It turned out that having data and code be the same thing was not nearly as helpful an AI technique as people hoped.

There’s an often cited article called The Lisp Curse, by Rudolf Winestock – written in 2011, the article posits that Lisp’s expressive power made it less likely to become popular, because:

Lisp is so powerful that problems which are technical issues in other programming languages are social issues in Lisp.

The idea is that it’s so easy to do complex things in Lisp that individual coders do them on their own, and more to the point that Lisp attracts programmers that want to do these things on there own, and so there’s no push to create community solutions that can be built on. Feels like the “my biggest weakness is I work too hard” of programming systems, but there’s probably something to it.

Ruby and Lisp

I think that what Ruby took from Lisp was more vibes than syntax. Concepts that seem important in Ruby and are also important in Lisp include:

• Everything is an expression, which is trivially true in Lisp, and an important part of Ruby
• Domain-Specific Languages, or the idea that you’d use metaprogramming to create a mini language that more clearly expresses intent. Lisp and Ruby do that in different ways, but there’s an underlying similarity of spirit.
• Lisp is also one of the few languages that differentiate strings than symbols.

I really do hope that you’ve enjoyed this series, it’s been a ton of fun to write, and it’s made we want to go back and play with these languages, and learn more languages. Basically, I love programming languages…