Codemirror 6 Experiments
When I found out that CodeMirror, the hugely popular in-browser code editor is was being re-written from the ground up in TypeScript with mobile usability as a priority, I got really excited. I did some experiments to try out the new version. This post is about these experiments. They are open source and available at https://github.com/datavis-tech/codemirror-6-experiments
These prototypes were useful in building the real-time features of vizhub.com.
Ubuntu Theme
As a first exercise to get familiar with the CodeMirror rewrite, I ported the Ubuntu-inspired theme from VizHub-UI to work with CodeMirror 6.
Developing this theme was fun! The biggest difference from CodeMirror5 is having to deal with ✨native text selections✨, which can be styled with ::selection. Using native selections (via contenteditable="true") is one of the things that really differentiates CodeMirror 6 from CodeMirror 5. This is what allows it to work well on mobile devices and integrate with the browser’s built-in text editing capabilities. More details about this in the CodeMorror 6 Design Document.
Mobile Support
I found that CodeMirror 6 did in fact work well on mobile! This is one of the things that excites me most about the rewrite. The ability to edit code on smartphones can open up the possibility of programming to a vast population of people, particularly in India where mobile computing dominates. When paired with CodeBoard, and Open Source Android keyboard with symbol and arrow keys required for coding, it actually seems plausible that one could write code on a smartphone using CodeMirror 6.
Server Rendering
On mobile, loading performance is key. I wanted to see if it was possible to render the critical components of the CodeMirror DOM and CSS on the server, so that when someone loads the page on mobile it displays something meaningful immediately, then is enhanced with interactivity after the JavaScript loads (which may take several seconds over 3G). I suppose you might call this sort of thing the beginnings of a “Progressive Web App”.
Even though CodeMirror is intended for use only in browsers, rendering its DOM on the server actually turned out to be possible. By leveraging JSDOM and doing some nasty tricks to set documenton the NodeJS global and mock out DOM APIs that don’t make sense on the server, I was able to get it to work!
The HTML (complete with classed elements for syntax highlighting!) and inlined critical CSS loads immediately (2KB). The editor is fully hydrated and interactive after the JS and CSS loads (342KB unminified). The CSS is bundled inside the JS using PostCSS, so the mobile page is fully operational after a grand total of 2 requests to the server.
Real-time Collaboration
I’ve always been a huge fan of real-time collaboration. Having everything synchronized across clients in real time just makes sense. Ideally everything should be synchronized, and this would be the “Web app paradigm of the future”, but the technical complexity of having everything synchronized is quite overwhelming. Especially if you also want other Modern Goodies like server rendering and client-side navigation. Way back in 2012 DerbyJS tried to be the end-all-be-all Web app framework with real-time everything, but it never really took off.
On many occasions I’ve found myself working with others on a code document, and having to describe the changes they need to make in order to fix their code or implement a feature. If only I could jump right into the code and edit it directly in real-time, and have my edits propagate to the original author immediately. It would revolutionize the remote tutoring experience, and make live remote pair programming a possibility! I do realize many products already have this feature (such as CodeSandbox), but the prospect of making it work with CodeMirror 6 is quite exciting.
I built a prototype of real time collaboration with CodeMirror 6 using ShareDB. ShareDB from what I can tell is the only solid Open Source implementation of OT (Operational Transformation) for arbitrary JSON documents as well as plain text. The technology is a bit old, but old as in finely aged wine not stale chips. ShareDB has been battle-tested over the years at Lever (its steward company) and elsewhere. ShareDB is a viable Open Source alternative to Firebase, which is great and all but reeks of vendor lock-in.
The first order of business for this experiment was to explore the CodeMirror plugin API. The CodeMirror 6 architecture is Redux-esque, where the state and view are totally separate, and changes are applied using “transactions” (akin to Redux actions). Every change flows through the system using unidirectional data flow. The system is composed of many “plugins” (akin to Redux reducers) that can intercept transactions. Plugins can also emit their own transactions, for example in response to user interactions.
This architecture is absolutely perfect for implementing an OT layer, because the transactions already encapsulate the differences in the text you’d want to send over the wire. What’s need to make the magic happen is a translation layer between CodeMirror 6 transactions and ShareDB-compatible JSON0 OT operations. This felt like a well isolated problem, so I put it in its own separate package codemirror-ot.
In order to get the ShareDB sample code to work in the modernized build environment used by CodeMirror (Rollup & Typescript), I had to jump through more than a few hoops. The default ShareDB package installation is badly outdated, as NPM does not use the latest versions prefixed with v1.0.0-beta. Frustratingly, ShareDB has been at 1.0 Beta for over 2 years now.
The WebSocket adapter for ShareDB is a separate package from ShareDB, and the original implementation is not maintained. So I used an active fork maintained by Teamwork. Teamwork also maintains an active fork of ShareDB itself, which I opted not to use (yet) as I don’t yet need any of its features that are different from the original ShareDB. Then I had to shim some Node builtins and globals to get the ShareDB client to work with a Rollup build, because the ShareDB client depends on EventEmitter and alsoprocess.nextTick.The ShareDB client is written under the assumption you’ll use Browserify as your build tool, but who uses Browserify anymore?
Anyway after much ado it worked! The working demo was the last big piece of the puzzle. This phase uncovered a slew of bugs with codemirror-ot, namely that transactions that don’t change text (such as cursor and selection movements) should not be translated to OT operations, and that a whole class of transactions in which multiple lines were inserted was not being handled correctly. Now those bugs have been squashed (code changes here).
The last phase of the real-time work was to make sure that CodeMirror was initialized to the content from the database, and that the ShareDB client handled all cases of synchronization properly. Several steps were involved here (code):
- Render the “snapshot” into a JSON string in the server-rendered HTML.
- Pass that “snapshot” into the ShareDB client to initialize the document.
- Use that “snapshot” to instantiate the CodeMirror contents.
Amazingly, this all appears to work. The following cases are handled:
- Typing before the JS is disabled, as the changes cannot be captured.
- Typing after the JS loads but before the WebSocket connection is established. ShareDB is smart enough to queue the changes, and send them when the connection is made!
- Typing in a remote client before the WebSocket connection is established. Again ShareDB is smart enough to know what updates to pull from the server when the connection is established (because it compares the “snapshot” version to the current version in the database and sends the diff).
A live demo of the editor with real-time sync is up at https://vizhub.io/ . Anyone can edit so I’m not sure what you’ll see when you open it. Try opening it in multiple browser windows side by side, or on your mobile device. Please do report any bugs you experience as issues.
Overall these experiments feels like a resounding success! I’m really looking forward to taking this work from experiment to prototype to product in building VizHub 2.0!
