A Future For Concurrency In Emacs Lisp

I recently wrote down my thoughts about Emacs’ concurrency efforts. In a way this post is the second part. While the first post went a long way to explain why I do not like threads this post tries to explain why I see futures as an easier, a faster way to add better concurrency support to Emacs.

Threads are just concurrency infrastructures—nothing that I’d like to use directly. But it’s not as if Emacs Lisp didn’t have infrastructure for concurrency already: We have asynchronous networking and asynchronous processes.

What we don’t have is a good API around these, and, more important, a ubiquitous model and interface for asynchronous operations.

What tools do we have to this end in Emacs Lisp? Just callbacks. More generally, CPS. But we know from other languages (Node.js, looking at you) that CPS leads to messy code when used at scale, and we see it in Emacs Lisp: Flycheck’s asynchronous code is hard to follow, because the inevitable excessive use of callbacks scatters it all over the place.

This CPS spaghetti makes asynchronous operations hard to use in Emacs Lisp, and—even worse—even harder to apply in hindsight. Just going from “call-process” to “start-process” requires a complete rewrite of the affected code in CPS.

We know better these days: We know monads, and we know that assuming a proper “do” notation lifting code from non-monadic bindings to monadic bindings is much closer to a purely syntactic transformation than rewriting bindings to CPS. In particular, we see that monadic code in a do-notation retains an important property of non-monadic code: It’s still local within a single expression and it’s still readable from top to bottom, i.e. in sequential form.

That’s not to say that Emacs needs monads—surely that’d be nice, but that’s not my point. We know that the idea of monads works well to model computations, whether we call them monads or not. Scala’s monads have “for” and “flatMap”, and are in a way “specialized” to the List monad. C# doesn’t even have proper monads but its “async”/“await” is just a specialized monadic syntax just for “Future”—C# calls them “Task”—monads.

We should use the same in Emacs Lisp: A monadic syntax—specialized for Futures if you want—to make asynchronous operations dead-simple to use, and easy to use in existing code. Take the following as an example:

(let* ((command (list "foo" ...))
       (return-code (apply #'call-process (car command)
                           nil nil nil (cdr command))))
 (when (= return-code 0)
   (with-current-buffer "*result*"
     (insert "some stuff"))))

And its asynchronous form:

(let* ((command (list "foo" ...))
       (process (apply #'start-process "foo" nil command)))
   (lambda (process state)
     (when (and (memq (process-status process) '(signal exit))
                (eq (process-exit-status process) 0))
       (with-current-buffer "*result*"
         (insert "some stuff"))))))

Wouldn’t it be so much better if we just wrote this instead:

 (let-async* ((command (list "foo" ...))
              (result (apply #'start-process-future nil nil command)))
   (when (and (memq (process-result-status process) '(signal exit))
              (= (process-result-exit-status result) 0))
     (with-current-buffer "*result*"
       (insert "some stuff")))))

And what’s more, with futures we had a common model for asynchronous operations to use everywhere.

As an example just think of “completion-at-point-functions” being able to return futures: We had asynchronous completion in Emacs Lisp. Wouldn’t even be hard to implement: Show synchronous completions right away, then await all asynchronous completions and re-render the completion buffer with all completions. No need even to change the current interface: Futures would fit in nicely without breaking compatibility as the calling code would just need to check whether a function returned a future or not.

That’s the main point of futures.

For sure Future can be implemented on top of threading, but they are so much more than just another API: They offer a ubiquitous interface for asynchronous results for use everywhere, right away.

And if we like to we can do threads and run futures on top of them. That’s just another advantage of futures: They don’t care for the underlying execution model, even be synchronous.

In a way futures are orthogonal to threads, but to me they are much more important than threads which are just another infrastructure in addition to async processes and networking that’s going to be under-used in the absence of a nice and simple concurrency API.

Now I’m biased by my own experience—I compare the ease of concurrency in Scala and Haskell with the hassle of even the simplest asynchronous operations in Emacs Lisp—but I know that threads won’t be of any use in Flycheck, but the futures would immediately make Flycheck’s code a lot simpler and allow us to make some operations concurrent that weren’t worth the complexity of concurrency with the current API (syntax checker predicates for instance).

I’m not saying that the efforts of the concurrency branch are wasted, or that we shouldn’t merge it when we get the chance, but I’m not sharing the community’s enthusiasm about it. Many of those who praise it and long for it haven’t yet considered the impact of threads on their code. I fear that threads come to Emacs with much ado only to starve to a niche existence as everyone realizes that they are terribly hard to use and come at great cost for existing code, leaving the problem of the blocking Emacs as large as it is just now.

We have more hope to solve that problem with futures which are easy enough for everyone to use concurrency. Concurrency for the masses, that’d be a thing in Emacs Lisp!