💦 clive: distributed spreadsheets of code+data

Everything that follows needs rewriting over better-defined abstractions.
Abstractions and their layering are emerging as follow:
- A Git over Xodus database.
- On top, auto-reloading classloader, w/ precise invalidation tracking
(not yet loaded ⇒ no reload, possibly in-place bytecode replacements).
- A functional object model, fully serializable and persistable in the database
  using Kryo, where entities can exist and evolve as references.
  Entities are maintained by arbitrary code through Git references and can contain
  full object graphs which can refer to other live references. Some references
  propagate their changes back to any consuming code.
  The full Git reference store is ACID.
  In that functional project model:
  - Trust/delegation models using SSH keypairs.
  - On top of said trust/delegation model, an execution dispatch model across services,
    including through P2P / clusters / cloud providers or config GUIs on laptops.
  - A distributed execution engine, which continuously rebuilds parts of the model
    as changes to their inputs are progatated.
    Amongst other tasks, that execution engine could, for example:
    - Start and stop other execution engines from anywhere in any DB over time,
      including as the result of tasks from other models,
    - Watch local filesystems and propagate changes into the object model,
    - Stream parts of the object model back to local filesystems,
      or various external services as they change,
    - Run arbitrary services, recurring tasks, etc.

What I think an adult unicorn looks like

Peaceful, specialized, living in big nerds bound in harmony by functional dynamic networks of trust

1. Clive is a distributed ecosystem of expressions for a Redux-like architecture (where the resolvers are replaced with Kotlin code built on top of layers of higher-and-higher level APIs) + React-like architecture (where the virtual DOM is replaced with Git objects and trees). After bootstrapping or seeding from an existing workspace, one drives and they feed each other. During those reaction chains, both arbitrary computations (such as code generation, compilation, packaging, code relocation, arbitrary execution all interleaved for environments like TypeScript where needed type definitions pour in and compilation doesn't depend on the source code itself) and their results can be cached, persisted and distributed programmatically.
2. Any directory with a Clive.toml becomes a Clive workspace. A living and breathing workspace started by a simple command line tool with zero dependencies, maybe from a $vcs clone to start from and alongside which to co-exist (with a simple ignore .clive/ as required integration).
3. In that workspace, remote and local code and data materialize by purpose; they can be modified, replaced and fetched, evaluated, propagated on the fly. For example one might want to start modifying just a couple of files in a published module, or fully replace it with a local checkout. Maybe they'd rather like to read some pre-built version from their vendor's store. Arbitrary local checkouts, arbitrarily attached to the workspace, can be manipulated with tools like version control systems, git included, without any disruption to the platform's live operation.
4. As anything changes, whether the current list of targets for the service (live-deploy this backend, run all the client tests against a local copy of this backend, etc.) or any of their transitive source materials (including local files monitored on the fly), all the corresponding state derivations and side-effects can be applied as fast possible, either by reading a local or remote cache (git over HTTPS and/or SSH), or by computing locally or remotely (remote Clive instance over HTTPS or SSH), from a long-lived service with hot caches and through long-lived connections and Git sessions.
5. The end result is soft-real-time code generation, builds, tests, publishing of artifacts, restarts of services in one's local debugger or development cluster, etc., from any change to some file on one's laptop, either offline or through a cloud worker, all within seconds for most small changes.
6. What's more, the full history of the workspace is recorded into sessions. Check out any point in time in a session and the full state might be ready in the local cache: the IDE instantly sees all its config, the generated code, compiled classes, fetched remote dependencies, pre-built node_modules hierarchies, failed compilations and test results as fast as the service can copy from this hot Git repository to its local filesystem.
7. A full history of developer sessions would be valuable as a cloud service; whenever the service on their laptop connects to the Internet, it automatically pushes all state transitions including file saves, streaming live when possible. This starting point would be quite close to the beginnings of real-time continuous integration and continuous deployment as a service. Secrets management, including for cloud clusters, would be one the interesting problems to address through Clive's highly dynamic nature.
8. Every workspace has an SSH private key associated with it in its VFS. This is a key starting point for most of the clustering and peer-to-peer networking functionality. That private SSH key could come from your Unix user, or derived from a config option in the VFS. It starts propagating into your developer session, and can spread to copies you personally trust. More keys can easily be inserted into the VFS through arbitrary logic which lets reactors maintain a set of active credentials on the fly for the systems which they access. At the networking level, those workspace SSH keys work to authenticate both clients and servers. But the trust delegation model is fully dynamic and the tricks for peer-to-peer don't satisfy all requirements for an enterprise setup today. For those environments, trust of GithHub Enterprise's SSH host key when fetching from repositories there can be provisioned, eg through shared plugin configuration in workspaces, and architectured around VFS derivation rule expressions. This incredible level of flexibility combined with strong conventions throughout the standard modules are the key to a secure distributed platform, where solid patterns for authentication and role delegation allow for fine-grained access control policies, a should-have for a professional tooling platform but unfortunately lacking today.

Inside most unicorns

1. A VFS reactor configured dynamically from the project reactor, hence fully programmatic, but with amongst other native features:
   1. Live writeback to the filesystem (mostly in .clive subtrees),
   2. Live ingest from the filesystem (excludes .clive everywhere),
   3. A mount-point system so a session could easily mount from a local copy of a module rather point to a remote and vice-versa,
   4. Overlay mounts, to override small parts of external projects for example;
2. A project reactor hooked to the VFS reactor with:
   1. Live modelization of the full project model in a single timeline;
   2. Functional model for the project, persisted as an immutable input-addressible object graph, LRU-persisted in the DB;
   3. A set of components to materialize the functional model into a filesystem, eg:
      1. Remote artifact fetchers,
      2. Code generators,
      3. Compilers,
      4. IDE configuration generator,
      5. Test executors,
      6. Service runners;
   4. “Targets” in the project functional model tracked as “alive” to drive execution of components, eg:
      1. Keep all generated code up-to-date for the IDE,
      2. Run client tests against the latest version of services,
      3. Live-deploy the last tested server to my cloud stack,
      4. On the cloud worker for master, live-publish static assets to S3 and invalidate CDN caches in soft-realtime;
   5. Powerful distribution models for the execution of some or all of those components, eg:
      1. Only run those tasks when a device is running on sector or is disconnected, otherwise distribute it to a local cloud of workers,
      2. From those, forward builds requiring a Mac operating system to a separate cluster of workers (useful to build iOS apps or Node native modules for example);
   6. Pretty strict conventions throughout, eg maximizing out-of-the-box integration in IDEA, mostly through config auto-generation within the Clive service rather than through an IDEA plugin, and carefully optimized behaviours as clive.kt is iterated on (including very eager dynamic mounting of import paths in the VFS, which can help with autocomplete and offering error checking instantly in IDEs and live service state views).
3. An SSH server offering a git-like interface, extended with session management;
4. An web server offering a rich UI, session management, a session browser and controller;
5. For clustered environments, Atomix-based coordination for parts of the VFS reactions.

// We parse imports to figure out which modules to bring in the classpath.
// For example this loads /thirdParty/clive.kt (or /thirdParty/ratpack/clive.kt if it exists)

import clive.model.jvm.artifacts.docker
import clive.model.jvm.artifacts.macosApp
import clive.model.jvm.artifacts.windowsSelfExtract
import clive.model.jvm.Dependencies
import clive.model.jvm.JVM
import clive.model.jvm.JVMArgs
import clive.model.kotlin.backendModule
import thirdParty.ratpack

backendModule {
  dependencies = Dependencies {
    compile {
      ratpack.classes
    }
    runtime {
      ratpack.transitive.exclude { thirdParty.hadoop }
    }
  }

  jvm = JVM {
    args = JVMArgs {
      memory = 16.GB
    }
  }

  artifacts = Artifacts {
    docker
    windowsSelfExtract
    macosApp(jvm = this@backendModule.jvm.copy(provider = JVM.Provider.Oracle))
  }
}