knowing what’s out there

I’m teaching a class to first-year college students. I just had a quick catch-up session with some of the students that had no prior programming experience, and one of them asked a fantastic question: “How do you know what library functions are available?”

In a classroom setting, teachers can work to prevent this kind of question by ensuring that students have seen all of the functions that they will need, or at least that they’ve seen enough library functions to complete the assignment.

But what about when they’re trying to be creative, and do something that might or might not be possible?

Let’s take a concrete example: a student in a music programming question wants to reverse a sound. How can this be done?

First thing, they probably look in the documentation for the sound library. They scan the list of function names, and none of them looks like something that reverses sounds. Should they give up?

Answer: no.

So what should they do?

Probably, they should look for a more general function that could be used to accomplish this task. For a first year student, this is a deeply daunting task. That is, they have to look at a sequence of thirty or forty functions and for each one, quickly determine whether or not it could be used in a program to accomplish the desired task. Even worse, it may be the case that they need to assemble several of these functions. In this case, the student needs to look at a function and think, “Gee… this function could be part of a solution, if I can figure out a way to solve this new problem that it creates.” You can see that this can lead to an exponential exploration of problem space.

Insted, an experienced programmer will probably tacitly generalize the desired function. “There’s no function that reverses a sound; is there any kind of more general sound rearranging that’s possible?” The answer here is yes. Unfortunately, there are actually two different ways to do this. First, there’s a clip function that can cut a portion out of a sound, and there’s an append operation that can glue sounds together. Separately, there’s a rearrange function that allows a user to provide a function to be used in mapping the frames of an old sound onto a new one.

Which one to use? It turns out that the first solution has some serious computational issues associated with it. If I want to reverse a three-minute sound, I’ll need to create a list containing 8,640,000 separate sounds.¹ This will probably take a while. The second one is much faster. So now, even after finding a set of library functions that work, the student needs to backtrack and try a different one.

Ouch.

Fast Forward One Day

Okay, so now it’s Tuesday.

I’m working on the handin server for my upper-division Programming Languages class, and I notice a funny problem in the logs. To wit:

[33|2017-10-03T11:05:03] checking Program2 for (clements)
[33|2017-10-03T11:05:04] running 794KB (170MB 184MB)
[33|2017-10-03T11:05:09] running 70MB (239MB 255MB)
[33|2017-10-03T11:05:09] done testing. 0 tests failed.
[33|2017-10-03T11:05:13] running 54MB (224MB 238MB)
[33|2017-10-03T11:05:17] running 54MB (224MB 238MB)
[33|2017-10-03T11:05:21] running 54MB (224MB 238MB)
[33|2017-10-03T11:05:25] running 54MB (224MB 238MB)
[33|2017-10-03T11:05:29] running 54MB (224MB 238MB)
[33|2017-10-03T11:05:33] running 54MB (224MB 238MB)
[33|2017-10-03T11:05:36] running 54MB (224MB 238MB)
[33|2017-10-03T11:05:40] running 54MB (224MB 238MB)
[33|2017-10-03T11:05:44] running 54MB (224MB 238MB)
[33|2017-10-03T11:05:48] running 54MB (224MB 238MB)
[33|2017-10-03T11:05:51] session killed (timeout) while running tests

What we see here is that at 11:05:09, the submission passed all the tests. However, at 11:05:51, the server timed out and the submission was marked as a failure.

Why?

The TL;DR here is this: I didn’t know (enough) about a library function. But it took me an hour to figure it out.

Here’s the extended version: I took a look at my checker module, and it looked fine. Specifically, it ended by logging the number of failures, and then showing a dialog box to the user that indicated the number of failures. Nothing wrong there.

I spent about ten minutes writing a careful e-mail to the Racket Users mailing list, trying to figure out what’s wrong with the handin server. As I was carefully explaining why it couldn’t be my fault… I realized that it was. Specifically, my dialog box was a modal one, waiting for the user to click ‘OK’. If the user doesn’t click ‘OK’, the call to the message box function doesn’t return, and the checker module doesn’t finish, and as far as the handin server is concerned, the user’s program has failed to halt and should be removed. Ouch. Delete e-mail before sending.

Side note: if you don’t use the unbelievably effective technique of carefully formulating a bug report for a group of people that you respect and whose time you’re leery of wasting—you should. It works.

Sub-side note: unfortunately, if you’re in college, you probably don’t have the judgment or patience to be aware that you’re wasting other people’s time. Ah well.

So, how to solve this problem? My first try is to put the call to message in its own thread.

I’m not entirely surprised when this starts behaving very badly indeed; I get messages like

[1|2017-10-03T11:10:20] ERROR: upload not confirmed: hekok

and

[2|2017-10-03T11:11:13] ERROR: upload not confirmed: chc

What the heck??

After a quick search for the string ‘hekok’ in the source, I decide that that’s probably a different scary bug, and that I don’t have time to debug it.

Side note: It makes me sad to see what I think might be bugs that I decide not to pursue; it’s a chance to make the world slightly better that I’m deliberately passing up. Of course, it might take a long time to track them down, and in many cases, they turn out not to be bugs at all.

So, maybe I can make this message part of a final result. Time to read some docs: I read the docs for the check: form that defines the checker module, and all of the optional arguments. Nothing. Then, on a whim, I decide to read the docs for the message form.

Aha! It turns out that the message form can be passed the style final, in which case the message is used as the final message to the student, after the submission is complete.

This is exactly what I want, and I use it.

Problem solved.

It’s at about this moment that I realize that what I’ve been experiencing is almost exactly the same problem that my students are facing; I don’t know what’s in the libraries. In some ways, I’m even more dangerous than the students, because I know of ways to hack around the problem, and solve it the wrong way.

So: how are we supposed to know what libraries are available? In my mind, it’s still a major open question. Let’s see if I can get anyone at RacketCon interested.

Alternate Ending

There’s an alternate, depressing conclusion to my experience: it’s incredibly hard to get software right. Every piece of software is riddled with problems like this that don’t occur frequently, and are arguably not even “bugs”, per se, except that they obviously are, and fixing one takes about an hour of skilled programmer time. There’s not enough time in the day. It’s all going to fall apart.

Or maybe we’re going to give up on “engineering” our programs and fall back to “evolving” them; we accept an ecosystem of horribly buggy software and choose the best stuff and apply weird patches and lash them together with baling wire. Nasty, but it really works.

¹ : … or 100x the number of seconds in a day. Coincidence, sorry.