ė - Goal and Overview

The goal of this project is to program computers using pluggable units of software.

To do this we need:

  • micro-concurrency
  • 0D
  • layers, relative sofware units
  • multiple languages.

Hello World

Very simple example

Leaf

Leaf

Container

Container

Re-Architecting

Container With Different Routing

Scalability

Pluggability is necessary for scalability, but, more elaborate (complicated) examples are needed.

Benefits

  • technical drawings come "for free"
  • concurrency comes "for free"
  • "build and forget" development
  • distributed programming comes "for free"
  • multiple-CPU paradigm
  • ability to plug together software components to create mimimal set of functionality

Key Insights

  • 0D - No Dependencies
  • FIFOs and LIFOs
  • Pipelines
  • Structured Message Passing
  • "First Principles Thinking"
  • Closures
  • "Parallelism" is composed of more than one concept

Approach

Formulate questions

Questions such as...

  • Why do hardware designs tend to work while software designs fail and become more complicated?

  • Pipelines are different from functions. How are pipelines different? 

  • Message passing. Is message-passing possible in the synchronous paradigm?

  • Message passing - asynchronous - has a bad rep because it is often ad-hoc. Is there an equivalent to "structured programming" for async message-passing?

  • Closures - are closures the same as "processes" in operating systems? 

  • DaS - Diagrams as Syntax. Why are most programming languages textual?

  • Tells - what is currently considered difficult? Multitasking, async, callbacks, mutation, sequencing, history, state ... Can these be improved? Are they difficult because they're difficult or because our notation makes them difficult?

  • Is "something" the same across all programming languages?

  • What is parallelism?

  • Operating Systems and Programming Languages were invented in the 1950's under the single-cpu assumption. Is the single-cpu assumption still valid?

  • Are end-users forced to use the "same" operating systems / computer environments as developers?

  • Can the goal (pluggable components) be sub-divided into smaller sub-goals?

    • Which properties must components have to be pluggable?
    • Which properties inhibit pluggability?

Synthesize

  • upon answering the above questions, it is possible to synthesize a new programming environment?

Tools / POCs

Tools and Proofs of Concept that are related to this project but not part of the 1-week Jam

  • https://github.com/guitarvydas/das2json
    • transpiles diagram testbench.drawio to JSON
    • transpiles diagram helloworld.drawio to JSON
    • uses hybrid diagram elements containing Python code
      • boxes, connections, ports written as a diagram
      • "Leaf" code written as Python (embedded in the diagrams)
      • the diagrams are hybrids - boxes + ports + lines + Python code
        • i.e. micro-concurrency written in diagram form, the rest written as regular Python code
        • do the minimum necessary, and let Python do the rest of the heavy lifting
    • uses PROLOG inferencing to create a factbase, including containment and connection relationships
    • could be simplified, probably with Ohm-JS
  • https://github.com/guitarvydas/das
    • similar to above, except slightly more ambitious
    • transpiles helloworld.drawio to Python, then runs the Python
    • transpiles d2py.drawio to Python, then runs the Python
      • d2py.drawio is a "script" for self-compiling the das-to-python transpiler to Python
      • the resulting Python script calls "make" for building the transpiler
      • the diagram shows one way to trap errors and quit
      • the UX (the diagram language syntax) could be more humane - at the moment, it consists of boxes and lines which might look "too complicated" to non-programmers (thought: maybe rely on box containment to "inherit" messages, eliding some of the ports and lines)
  • https://github.com/guitarvydas/vsh
    • checkout afa53b8
    • contains very old (pre-2013) Visual SHell experiment
    • diagram compiler written in itself (using yEd editor)
    • generates .gsh files
    • .gsh is grash a minimal (approx 8 instruction) assembler for invoking UNIX C system functions (e.g. dup2(), exec(), etc.)
    • (I ran out of time during this Jam to make this work again)
  • pipelines
    • see UNIX shells
    • see Aho's paper on using pipeline syntax as a replacement for Denotational Semantics syntax
    • (I ran out of time to dig up this reference during this Jam)
  • micro-concurrency
    • partially based on implementing Harel StateCharts
    • partially based on Holt's book Concurrent Euclid, UNIX, Tunis
    • partially based on designing hardware using state machine diagrams
    • "sequencing" and "concurrency" is "easy" with state diagrams, less easy with functional notation
  • conclusions about 0D and "parallelism"
  • HSM - Hierarchical State Machines

Status

This project is essentially about joining the pieces of the project into a coherent whole.

  • Most of the pieces are on the floor (see sub-projects in detailed documentation) and getting simpler with each iteration.
  • Currently working on Python version of Leaf components using HSMs (hierarchical state machines), trying to simplify HSM P.O.C.

much more detail, discussion, references

Recent Activity

update Oct. 24, 2022

This diagram:

Hello World

is transpiled to this JSON:

[
  [
    {
      "children": [ {"kind":"Hello", "name":"cell_7"},  {"kind":"World", "name":"cell_8"} ],
      "connections": [
	{
	  "receivers": [ {"receiver": {"component":"cell_7", "port":"stdin"}} ],
	  "senders": [ {"sender": {"component":"cell_6", "port":"stdin"}} ]
	},
	{
	  "receivers": [ {"receiver": {"component":"cell_8", "port":"stdin"}} ],
	  "senders": [ {"sender": {"component":"cell_7", "port":"stdout"}} ]
	},
	{
	  "receivers": [ {"receiver": {"component":"cell_6", "port":"stdout"}} ],
	  "senders": [ {"sender": {"component":"cell_8", "port":"stdout"}} ]
	}
      ],
      "id":"cell_6",
      "inputs": ["cell_17" ],
      "kind":"HelloWorld",
      "name":"HelloWorld",
      "outputs": ["cell_15" ],
      "synccode":""
    }
  ],
  [
    {
      "children": [],
      "connections": [],
      "id":"cell_7",
      "inputs": ["cell_12" ],
      "kind":"Hello",
      "name":"Hello",
      "outputs": ["cell_10" ],
      "synccode":""
    }
  ],
  [
    {
      "children": [],
      "connections": [],
      "id":"cell_8",
      "inputs": ["cell_11" ],
      "kind":"World",
      "name":"World",
      "outputs": ["cell_14" ],
      "synccode":""
    }
  ]
]

and the above JSON is transpiled to this Python:

from message import Message
from sender import Sender
from selfsender import SelfSender
from receiver import Receiver
from selfreceiver import SelfReceiver
from upconnect import UpConnect
from downconnect import DownConnect
from routeconnect import RouteConnect
from passthroughconnect import PassThroughConnect
from container import Container
from Hello import Hello
from World import World
class HelloWorld (Container): 
  def __init__ (self, parent, name):
    cell_7 = Hello (self, f'{name}-Hello-cell_7');
    cell_8 = World (self, f'{name}-World-cell_8');
    self._children = [cell_7,cell_8]
    self._connections = [
      DownConnect (SelfSender (self,'stdin'), Receiver (cell_7,'stdin')),
      RouteConnect (Sender (cell_7,'stdout'), Receiver (cell_8,'stdin')),
      UpConnect (Sender (cell_8,'stdout'), SelfReceiver (self,'stdout'))
      ]
    super ().__init__ (parent, name, self._children, self._connections)

and is transpiled to Common Lisp

(in-package "EH")
(defun new-HelloWorld (parent name)
  (let ((self (make-instance 'Container :parent parent :name name)))
    (let ((cell_7 (make-instance 'Hello :parent self :name (format nil "~a-~a-~a" name "Hello" "cell_7"))))
      (let ((cell_8 (make-instance 'World :parent self :name (format nil "~a-~a-~a" name "World" "cell_8"))))
	(let ((children (list cell_7 cell_8 )))
	  (let ((connections (list  
			      (make-instance 'DownConnect :sender (make-instance 'SelfSender :component self :port "stdin") :receiver (make-instance 'Receiver :component cell_7 :port "stdin")) 
			      (make-instance 'RouteConnect :sender (make-instance 'Sender :component cell_7 :port "stdout") :receiver (make-instance 'Receiver :component cell_8 :port "stdin")) 
			      (make-instance 'UpConnect :sender (make-instance 'Sender :component cell_8 :port "stdout") :receiver (make-instance 'SelfReceiver :component self :port "stdout"))  )))
	    (setf (children self) children)
	    (setf (connections self) connections)
	    self))
	))))

Wart

Branch master has a wart - a hard-coded path (see README.md). This will be fixed in branch dev.

Next Steps

Cleave ė into multiple parts

  • diagram to JSON component descriptors
  • JSON component descriptors to Python
  • JSON component descriptors to Common Lisp

Create spin-off which enables manual creation of ė and HSM and 0D code in Python.

  • diagram transpiler - copied das as subdirectory into project

  • hw.drawio -> hw.json transpile helloworld diagram

  • hwhw.drawio -> hwhw.json transpile re-architected diagram

  • wart: das does not transpile simple hello.drawio diagram -> just use hw.drawio and hand-carve the generated json out of it (or just continue ignoring the issue, since hello.drawio is very, very simple and not worth any trouble)

  • diagram parser: das.ohm parses output hw.json and hwhw.json

  • next

    • create .fmt for diagram parser and emit same .py code that was written manually

git (same place as before)

update: 2022-08-24

make all now runs 4 tests.

Test 4 contains 2 world.pys and produces {'stdout': ['hello', 'world', 'hello', 'world']}

Test 4 shows off some deep technical aspects: fan-out, fan-in, punting messge from Container to Child, Child output routed to other Chidren.

Test 3 is test2 wrapped in another layer, just to test whether it can be done.

Also fixed Hello.py->hello.py case insensitivity in MacOS. Make was failing because it could not find hello.py, whereas other tools (like vscode) did not fail on this.

Next: create .drawio diagrams and transpile them to the above .py code (diagrams as shown in the documentation. [Intend to use https://github.com/guitarvydas/das]

branch: master

(day-after jam, added Usage and Makefile to branch "master")

Usage

make

This runs run.bash which runs a single 0D Leaf component echo.py and prints its output queue at the command-line.

Test.py invokes hello.py and feeds it a trigger message (True).

Then test.py invokes world.py and feeds it the above output.

World.py is almost like hello.py except that hello.py does not echo its input whereas world.py does echo its input. World.py emits 2 outputs for each input it receives.

Both components - hello.py and world.py send outputs to their respective output queues.

The final result is:

  1. hello.py outputs ['hello'] on output port 'stdout'
  2. world.py inputs 'hello' on its input port, then outputs ['hello', 'world'] on its output port 'stdout'.

Test.py invokes the two components and sends them messages in sequence. This process can be generalized to what I call a Container component, but, I didn't get there before the jam ended.

Note that the .outputs () method returns a dictionary of stacks (LIFO) of values for each port. This was a conscious decision. LIFOs, Alists are what original McCarthy FP-style code used. Sector Lisp continues the tradition of using LIFOs. I think that this is one of the secret ingredients for the anti-bloatwareness of Sector Lisp. No RAM, therefore, no heaps, therefore, no mutation, therefore, simplified data access via push/pop/lambda-binding. Lambda-binding and LIFO call-stacks fit together to make small code and no-fuss structure-sharing.

Sequencing in this paradigm is explicit and caused by the order of the sends. Sequencing in most textual programming languages is implicit and is controlled by the syntax of the language (lines of code are ordered sequences).

End of Jam

The jam ended before test.py worked correctly, but, today - 1 day after the jam - test.py is working.

Post Jam

Next, would be to make a Container (Composite) component - helloworld.py that contained two components that can be chained together. Chaining is not necessary in this paradigm and I keep it only to make the examples look more familiar.

After that would come a rearrangement of helloworld.py that would contain one hello.py and two world.pys, resulting in "['hello', 'world', 'hello', 'world']"