Jolt внедрил сериализацию образов программ и переносимый слой Scheme

Program images and portable Scheme backends for Jolt Two features landed in Jolt recently, both of which fell out from loosening the coupling between Jolt and the host runtime. The first is the ability to serialize program images in the style of Common Lisp and Smalltalk, and the second is to have a portable Scheme layer decoupled from the Chez runtime. Let's take a look at what these things buy Jolt in practical terms. Images: a black box for your program If you've ever had to support a production system like a web application, then you know the value of having good logging. What typically happens is that you sprinkle statements through the code, ship them somewhere searchable, and then use them as bread crumbs when something breaks. When a system has errors, you have to first reconstruct what it was doing to understand what happened. If your logging didn't capture a critical piece of information, then the investigation devolves into guess work and attempts to reconstruct the state which caused the error. This can be a particularly frustrating experience when production goes down at 3am. The trouble with logging is that it forces you to guess the question before you know it. Each log line is a projection of program state chosen in advance, often being composed of two or three things that seemed relevant when you wrote the call. If the actual cause happened to be in the fourth thing, the log tells you nothing of value. You can't go back and ask a different question, because the values are gone, and you likely have no way to access them even if they're still in memory. All you have to work with is the rendering you decided on when you wrote the code originally. As a result, people tend to over-log defensively, which creates a different type of problem where you end up with a volume of logs that add noise and make tracing harder while often still missing the fields that actually mattered. But what if I told you that there was a better way, and you didn't have to rely on logging at all? This is precisely what jolt.image lets us do. Instead of choosing what to record ahead of time, you can just dump the whole state of the program at the time of the error to disk. Then you can just copy the file to your local machine, load up the state in the REPL and then poke around in it to see what happened. Here's what that looks like in practice. To dump the state when an error happens, all you have to do is call image/dump! in the exception handler: (try (process-batch! batch) (catch Exception e (image/dump! (str "crash-" (random-uuid) ".jimg") {:error (Throwable->map e) :batch batch :pending @work-queue}) (throw e))) Or skip the enumeration entirely and take the whole program: (catch Exception e (image/dump-world! "crash.jimg") (throw e)) dump-world! walks the var table and writes every data var's root so that nothing in your code has to declare what its state consists of up front. The image is architecture agnostic, so an image written on an arm64 server will restore fine on your x86-64 desktop. Once you've copied the file to your local machine, you just have to open it in a REPL: $ jolt repl user=> (require '[jolt.image :as image]) user=> (image/restore-world! "crash.jimg") 412 user=> (filter #(nil? (:price %)) @app.core/current-batch) ({:id 4182, :sku "B-77", ...}) What comes back are the values that were present in memory at the time the problem occurred. Maps, records, cycles and shared structure are all intact with their respective metadata attached, and functions come back callable as well. A named function resolves to the live one, while an anonymous closure travels as its source form along with its captured values to get compiled back on restore. So you can actually call the function that failed, on the data that failed in the REPL and see exactly what went wrong. You can now ask a strictly larger set of questions than any log file can answer, and you didn't have to know any of them in advance. You can think of the program image as a black box recorder. Just like an aircraft stores the instrument states to allow investigators to decide afterwards what to look at, a program image gives you all the information needed to debug the problem. At this point, a keen reader might ask how this approach handles open resources such as a socket or a file port that can't be serialized. The approach I landed on was to have dump-world! write them as stub records by default. Once the image is restored, you can list them by calling (image/stubs), and either register a resolver that reopens them or swap live values in by hand from the REPL using (image/register-stub-resolver! kind-or-pred f). One limitation is that a closure over a compile-time constant refuses to dump because the constant gets folded into compiled code and can't be recovered while the stored source still needs it. Closures built by partial and comp have the same problem for a related reason. The image's header is also checked on read so that you don't get stale data from an incompatible build. While dump-world! will dump everything it can, dump! is strict by default, and names the path to the offending object instead of writing something subtly incomplete. It's useful in cases where you want to be explicit about having the whole state available. Sending a program to someone While post-mortem debugging is an obvious use case, another interesting application is to facilitate collaboration. An image is a program state you can hand to another person which opens up some intriguing possibilities. For teaching, that means you can put someone directly into the middle of a running system, with real data loaded, without them having to build or seed anything. Imagine being able to say "here's the image with the pipeline implemented halfway through; go look at stage 2 in it." For a bug report, it means a colleague can reproduce your problem exactly, because they are just loading your state. They can poke at it, change something, dump it again, and send it back to you. It's a whole different relationship with a program than what we're used to since, traditionally, source code is treated as the artifact that we pass around. Jolt moves the needle closer to how Smalltalk and the Lisp machines worked, and how save-lisp-and-die still works in Common Lisp today. The program is a live thing you keep that can evolve over time as opposed to being a recipe you can run. Durable session state Another interesting application would be use cases such as desktop apps, where the user can save their session and then get back to where they left off when restarting the app. I built a TodoMVC example illustrating what that looks like in practice. You can click around, add or modify tasks, then dump the state and reload it in a fresh instance. Portable Scheme backends Originally, my goal for Jolt was to build a Clojure implementation on Chez Scheme. However, Jack Rusher pointed out that it would be possible to factor out a portable runtime and compiler making Chez just one target among several. Gambit was the obvious choice for a second host since it has a JavaScript backend making it possible to run Jolt in a browser. If you visit the official site you can now try playing in an interactive REPL, which is Jolt running in the browser. The refactor split the Scheme part of the compiler into three distinct layers. The core is written in portable Scheme implementing collections, sequences, the reader, the printer, vars, multimethods. This is ordinary Scheme that any serious implementation runs unchanged. Next there is the adapter contract where the host shows through. Every host capability goes through an sa-* entry point, and the contract file lists 72 names grouped into tiers: system (clocks, environment, exit), threads, eval, introspect (continuation frames for backtraces), ffi, native-compile, and image. A target can either implement a tier or degrade it honestly so that an absent capability raises a message-carrying error or returns empty. That last property is key for making partial ports actually usable. The Gambit version runs with ffi, native-compile and image degraded, and declares its capabilities explicitly. Finally, target-owned files are the two pieces nobody can share. These include the adapter itself, and the hash kernel. For example, the Chez version uses unsafe fixnum operations that other Schemes spell differently. On the compiler side, per-target differences go through a primitive table with the main entry being the unsafe-op prefix, so a target that maps it to the empty string simply gets checked operations everywhere stating whether it's safe, portable, or slower. The dialect-specific work is both smaller than you'd guess and duller than you'd hope. Most of it involves mapping records to their parent types, the hashtable API, fx operation spellings, the shape of error objects, and making the hash function produce bit-identical output to Chez. The Gambit port weighs in at about 6,000 lines, and a good fraction of it, including the seed itself, is generated on Chez rather than having to be written from scratch. Cross-minting the seed from a known working build is the trick that keeps a new target from having to bootstrap itself. Why more than one Scheme is worth having The immediate payoff is reach. Gambit compiles to a single JavaScript file, so Jolt now runs in a browser allowing for a REPL on the front page of the site using the real compiler and standard library evaluating directly on the client. It's not terribly fast, but works for a demo. Scheme is a whole family of languages with different dialects each optimizing for different use cases. So, the deeper payoff here is in opening up an ecosystem of implementations that made different bets. All of them share core language semantics, but each dialect puts its own twist on the language providing a runtime optimized for different use cases. Jolt can now piggyback on this whole ecosystem providing a Clojure layer on top. Chez makes an excellent default since it's fast, relatively small, feature-rich, with real threads, an FFI, and native compilation. Gambit gets you to JavaScript and C. Meanwhile, a whole-program optimizing compiler in the Stalin lineage is a different proposition entirely; it affords aggressive closure and type analysis producing tiny output that suits a small binary shipped to a constrained device where startup and footprint dominate. Such a compiler typically has no runtime eval at all, which sounds disqualifying until you notice that the seed is already cross-minted on Chez. This way the compiler can live on one Scheme while the emitted program runs on another. The key part here is that the program is the same Clojure regardless of which host you target. What changes are the capabilities available, which have to be stated in the contract providing clear boundaries for what can be expressed by each runtime. Thanks to many existing Scheme implementations, the same code can now run on a server, in a browser tab, as a tiny static binary, or get embedded in existing programs. A program shouldn't be trapped in the process that started it, nor should it be married to the runtime it was first compiled for. Lisps were always meant to be flexible, and Jolt embraces this philosophy.