LOG.md

## Ponyo Dev Log

**2019 10 21**
Today I think I will take my time on the compile-time build/sys for Ponyo. Answering questions like:
- how do I \#define classes so that they don't compile when I build an instance for a particular device?
- how do I make those pull through to the system-report ... ?
- same config. for class related interrupts ... only declare the interrupts if the class is being compiled, der
- *but then* these classes are kind of always-on ? we can maybe ignore this for now.

To start, I would have some simple define option to declare whether / not a thing is going to compile. I guess experiments for this I should run with something new ... A hardware driver for the lights panel, maybe? An easy example at least.

OK: the trouble is in class declarations. Including / or not / I need to invent some structure here.

I suppose ideally there would be an easy way to include in the build...

#define HUNK_TO_INCLUDE_DEF "string/path/hunk/identifier"
...

top-of-hunk is like

#include "buildconfig.h"
#ifdef HUNK_TO_INCLUDE_DEF
... same as usual,
#endif

and adding to the system... have to have some kind of list?
would like to check what preprocessor moves are available.

Here I find myself wanting something like the string literal in js... or some compile-time scripting. Instead I think I have to carefully consider this graph...

What constitutes success? I'd like to call this daylong project when I can:
- one-word define (1) a list of 'circuit-specific hunks' to add, and (2) a bootstrap program
- that (1) should add to the list and, probably more difficult, to the compile...

OK: I think I am at the heart of this issue. I want a kind of 'build list' item, this is a .h file, that #includes all of the necessary hunks, and runs the `allocateHunkByType(String type)` function, that returns the dummy Hunk* that the manager needs. Yeah, ok, this just means I need an intermediary... and some way of writing / handling a string list at compile time. Oboy.

OK: I have something running, it's a bit cumbersome and maybe in the future it can get better, alas... Oh, another idea. Although, tough.

...
static members? the uid array & startup routine ? use lldebug.

This is more difficult than I thought. Typed languages, hard! The trouble is in allocating the different types; writing that 'new' call - I have to do that with some #define logic, otherwise I have to compile every single type every time, and that's no fun (and takes up memory!). At the moment, I have to add new modules in like 5 different places. I can maybe check out how to do this properly with smoothieware... or ? just bite that bullet, maybe ... this might be a post-week-in-cpp-world question.

To finish:
- have / should do some minimal sorting at ponyo init, can take that string list and cull empties, write a global value for the size of the list.

Ha! Yeah, ok, should commit (haha) on this list thing, uids, and see if I can use static class members to handle the string-naming side of it?

Sure, so this is *just fine* at this point, although I have to touch a few spots to write / add a new hunk. To spec, I'll try that, for the board-w-many-lights.

**make the files**
I'll start by making my cpp files: I'll do `hunks/drivers/tlc5940.h` and `hunks/drivers/tlc5940.cpp` for this thing.

**the includes**
In the top of my `.h` file for this new thing, I include the build list, then I can ignore the whole thing at compile time if it's not spec'd for the board:

```cpp
// header
#ifndef SPI_TLC5940_H_
#define SPI_TLC5940_H_

#include "build_list.h"

#ifdef BUILD_INCLUDES_HUNK_DRIVER_SPI_TLC5940

// code will live here...

#endif
#endif
```

```cpp
// cpp
#include "spi_tlc5940.h"

#ifdef BUILD_INCLUDES_HUNK_DRIVER_SPI_TLC5940

// same

#endif
```

Then I can include it in a board, in my `build_list.h`

```cpp
// pick 'yer board
//#define BOARD_IS_STEPPER
//#define BOARD_IS_ROUTER
#define BOARD_IS_DLPCORE

#ifdef BOARD_IS_STEPPER
  #define BUILD_INCLUDES_HUNK_STEPPER
#endif

#ifdef BOARD_IS_ROUTER
  #define BUILD_INCLUDES_HUNK_COBSERIALB
  #define BUILD_INCLUDES_HUNK_COBSERIALC
  #define BUILD_INCLUDES_HUNK_COBSERIALD
  #define BUILD_INCLUDES_HUNK_COBSERIALE
  // #define BUILD_INCLUDES_HUNK_COBSERIALF
#endif

#ifdef BOARD_IS_DLPCORE
  #define BUILD_INCLUDES_HUNK_DRIVER_SPI_TLC5940
#endif
```

This is picked up in `build_bootstrap.h`, where I do:

```cpp
#ifndef CIRCUIT_H_
#define CIRCUIT_H_

#include "build_list.h"
#include "hunks/hunks.h"
#include "manager.h"

// ------------------------------------------------------- //
// ------------------ DEFAULT HUNK LIST  ----------------- //
// ------------------------------------------------------- //

#include "hunks/hunk_link.h"
#include "hunks/comm/hunk_cobserialusb.h"
#include "hunks/comm/hunk_cobserialrj45_a.h"
#include "hunks/control/hunk_saturnjog.h"
#include "hunks/math/hunk_adder.h"
#include "hunks/math/hunk_booleaninversion.h"
#include "hunks/flow/hunk_filter.h"
#include "hunks/flow/hunk_accumulator.h"

#ifdef BUILD_INCLUDES_HUNK_STEPPER
  #include "hunks/hunk_stepper.h"
#endif

#ifdef BUILD_INCLUDES_HUNK_COBSERIALB
  #include "hunks/comm/hunk_cobserialrj45_b.h"
#endif

#ifdef BUILD_INCLUDES_HUNK_COBSERIALC
  #include "hunks/comm/hunk_cobserialrj45_c.h"
#endif

#ifdef BUILD_INCLUDES_HUNK_COBSERIALD
  #include "hunks/comm/hunk_cobserialrj45_d.h"
#endif

#ifdef BUILD_INCLUDES_HUNK_COBSERIALE
  #include "hunks/comm/hunk_cobserialrj45_e.h"
#endif

#ifdef BUILD_INCLUDES_HUNK_COBSERIALF
  #include "hunks/comm/hunk_cobserialrj45_f.h"
#endif

#ifdef BUILD_INCLUDES_HUNK_DRIVER_SPI_TLC5940
  #include "hunks/driver/spi_tlc5940.h"
#endif

#define HUNK_LIST_LENGTH 64

String hunklist[HUNK_LIST_LENGTH];
uint16_t hlc;

Hunk* allocateHunkByType(String type){
  // I *dont* have a good answer for this yet,
  // I'll run this code many times, it's inefficient, that's fine for now !
  // if you are a cpp expert, tweet @ me for a solution
  hlc = 0;
  Hunk* ret = nullptr;
  hunklist[hlc++] = "link";
  if (type == "link"){
    ret = new Link();
  }
  hunklist[hlc++] = "comm/COBSerialUSB";
  if (type == "comm/COBSerialUSB") {
    ret = new COBSerialUSB();
  }
  hunklist[hlc++] = "comm/COBSerialRJ45_A";
  if (type == "comm/COBSerialRJ45_A") {
    ret = new COBSerialRJ45_A();
  }
  hunklist[hlc++] = "control/saturnjog";
  if (type == "control/saturnjog") {
    ret = new SaturnJog();
  }
  hunklist[hlc++] = "math/adder";
  if (type == "math/adder") {
    ret = new Adder();
  }
  hunklist[hlc++] = "math/booleaninversion";
  if (type == "math/booleaninversion") {
    ret = new BooleanInversion();
  }
  hunklist[hlc++] = "flow/filter";
  if (type == "flow/filter") {
    ret = new Filter();
  }
  hunklist[hlc++] = "flow/accumulator";
  if (type == "flow/accumulator") {
    ret = new Accumulator();
  }
  #ifdef BUILD_INCLUDES_HUNK_STEPPER
    hunklist[hlc++] = "stepper";
    if (type == "stepper"){
      ret = new Stepper();
    }
  #endif
  #ifdef BUILD_INCLUDES_HUNK_COBSERIALB
    hunklist[hlc++] = "comm/COBSerialRJ45_B";
    if (type == "comm/COBSerialRJ45_B"){
      ret = new COBSerialRJ45_B();
    }
  #endif
  #ifdef BUILD_INCLUDES_HUNK_COBSERIALC
    hunklist[hlc++] = "comm/COBSerialRJ45_C";
    if (type == "comm/COBSerialRJ45_C"){
      ret = new COBSerialRJ45_C();
    }
  #endif
  #ifdef BUILD_INCLUDES_HUNK_COBSERIALD
    hunklist[hlc++] = "comm/COBSerialRJ45_D";
    if (type == "comm/COBSerialRJ45_D"){
      ret = new COBSerialRJ45_D();
    }
  #endif
  #ifdef BUILD_INCLUDES_HUNK_COBSERIALE
    hunklist[hlc++] = "comm/COBSerialRJ45_E";
    if (type == "comm/COBSerialRJ45_E"){
      ret = new COBSerialRJ45_E();
    }
  #endif
  #ifdef BUILD_INCLUDES_HUNK_COBSERIALF
    hunklist[hlc++] = "comm/COBSerialRJ45_F";
    if (type == "comm/COBSerialRJ45_F"){
      ret = new COBSerialRJ45_F();
    }
  #endif
  #ifdef BUILD_INCLUDES_HUNK_DRIVER_SPI_TLC5940
    hunklist[hlc++] = "driver/spi_tlc5940";
    if (type == "driver/spi_tlc5940"){
      ret = new SPI_TLC5940();
    }
  #endif
  return ret;
}

// ------------------------------------------------------- //
// ---------------------- BOOTSTRAPS --------------------- //
// ------------------------------------------------------- //

// ok, here is ah board definition:
// since we define a global 'bootstrap' fn,
// we should be blocked from defining multiples of these

// to start a test, I'll see how I might just include the stepper when this is
// defined

#ifdef BOARD_IS_STEPPER
  Stepper stepper; // = new Stepper();

  void TC0_Handler(void){
    // stepper ...
    stepper.isr_handler_a();
  }

  void TC5_Handler(void){
    // etc
    stepper.isr_handler_b();
  }

  void bootstrap(Manager* p){
    // and bootstrap,
    p->addHunk("link");
    #ifdef BOOT_USB
      p->addHunk("comm/COBSerialUSB");
    #else
      p->addHunk("comm/COBSerialRJ45_A");
    #endif
    // (0) manager to (1) link's 1th
    p->addLink(0, 0, 1, 1);
    p->addLink(1, 1, 0, 0);
    // (1) link's 0th to cobserial (2) 0th
    p->addLink(1, 0, 2, 0);
    p->addLink(2, 0, 1, 0);
    // more plus
    stepper.name_ = "step_driver";
    p->includeHunk(&stepper);
  }
#endif

#ifdef BOARD_IS_ROUTER
  //
  void bootstrap(Manager* p){
    // and bootstrap,
    p->addHunk("link");
    #ifdef BOOT_USB
      p->addHunk("comm/COBSerialUSB");
    #else
      p->addHunk("comm/COBSerialRJ45_A");
    #endif
    // (0) manager to (1) link's 1th
    p->addLink(0, 0, 1, 1);
    p->addLink(1, 1, 0, 0);
    // (1) link's 0th to cobserial (2) 0th
    p->addLink(1, 0, 2, 0);
    p->addLink(2, 0, 1, 0);
    // more plus
  }
#endif

#ifdef BOARD_IS_DLPCORE

  // assuming I'm going to want some interrupts for this build:

  SPI_TLC5940 tlcdriver;

  void bootstrap(Manager* p){
    // and bootstrap,
    Hunk* lnk = p->addHunk("link");
    // *whistles*
    State* swapOption = new State_string("outputList", "mgrMsgs (byteArray), brightness (uint32)", 512);
    lnk->states[3]->swap = swapOption->chunk;
    lnk->stateChangeCallback_3();
    #ifdef BOOT_USB
      p->addHunk("comm/COBSerialUSB");
    #else
      p->addHunk("comm/COBSerialRJ45_A");
    #endif
    // (0) manager to (1) link's 1th
    p->addLink(0, 0, 1, 1);
    p->addLink(1, 1, 0, 0);
    // (1) link's 0th to cobserial (2) 0th
    p->addLink(1, 0, 2, 0);
    p->addLink(2, 0, 1, 0);
    // add our driver, and hookup...
    tlcdriver.name_ = "led_driver";
    p->includeHunk(&tlcdriver);
    p->addLink(1, 2, 3, 0);
    // more plus
  }
#endif

// here's a problem: if it's boot-usb, but we'll also
// want to have cobs_a available ... ? how to handle interrupts, then ?
// basically have to dynamic allocate.
// regardless, this doesn't have to be the master-blaster yet, just make some
// machines jog ...

/*
#include "hunks/pins/hunk_pin_localring.h"
#include "hunks/pins/hunk_pin_pa10_input.h"
#include "hunks/pins/hunk_pin_pa12_output.h"
*/

/*
#include "hunks/hunk_loadcell.h"
*/

#endif // end CIRCUIT_H_
```

Great, that's 'fine'.

**2019 10 13**

Lots of this world is annealing. In particular, the note below: static code for particular hardwares, focus on the allowance of specialization of firmwares rather than a one-code-to-kill-them-all. Until we go FPGAs all the way down, it's not worth the constraint.

**2019 09 19**

Back at this, and working on build / config systems so that I can implement good DMA systems, interrupts, etc, for specialized hardware. Byte code config. Const class members, some single list of 'registered' classes, some always-on (low-level hardwares), etc.

**2019 07 19**

ok, *(1st) note is that I should be logging dates for notes more often... (2nd) note is that I am just now through sending some bytes across the network (finally) and ponyo's guts are more or less complete (just not that many types available yet). hopefully today I will be working through testing each PHY on the router, and then up to deploying big systems, i.e. moving the littlerascal about.

**previously**

for tracking, you're up to transporting something through cobserial->link->themanager. before completing this loop, you need to formulate real messages. that means logging through cuttlefish and nautilus, to get to a hello, then just follow that sequence. some typeset, eventually the hunk serialization monster (soon).

hanging in the wings is the link's updating operations, as well as the heavy serialization work, and some example of states being set and updated. otherwise, seems like most of the buttons are in order and it's just work.

also pending is this work for multi-ported (?) managers - this feels something like just an idiosyncracy of a multi-link'd manager, but this would also be true of, say, a server or ? want it for the fyphy & the usbphy combo.

ok, cool, worked through a series of type / set / issues

towards a proper link (and same str. will exist for manager) we need some small local buffer, that can store multiple messages. this is the outgoing side of the link. need to be able to hello() and ack() reliably, as well as send msgs.

so, some memory superstructure that does a put(from, bytes) ... ooor ... how to track? this is a trick. observe the other side link (js) ... see how often, how many can be expected. as I beleive, it's as many as the whole set of inputs or outputs? make the buffer huge, take for granted small messages?

2019-06-27

OK, have had this success:

![hello-cuttlefish](ponyo-hello-cuttlefish.png)

Then, I have to:
 - reply with a list (of things that can be added)
 - add one new hunk: adder, w/ state var to add...
 - respond to adding that hunk,
 - js number -> and logger to pdevtwo system, thru below
 - respond to link addns, link rmval
 - to link rms
 - to state updates
 - evaluating hunks
 - name changes (nice but not necessary)
 - etc

Top notes from writing session:
 - 'state is the only obj that doesn't get/set like a data port' is something you knew but didn't think clearly. this has implications for the way you (re?) write the implementation ... (1) radical re-write would be to present it as another check-if-you-will gateway, responsibility of loop to set. (2) - in any case, do this - set rules for default state action: does it get set, or do we *have to* explicitly write a handler? probably auto set: default to action. how do we make sure that cpp implementation doesn't call uninitialized fn pointers?
 - transport doesn't need to be local to nets: just go by len, len always static for known types ... what was that about?
 - I think you've got to write this new typeset / struct thing? might only take 30 mins ... of focus ... use the zen, the coffee, the morning

Following a completely-alive Ponyo, I'd love to do a code review, work through that typing system, and write it up (some). Go back to normalized names for hunk classes / input output ... i.e. ->name_ or ->name, pick one. Not private, so.

These are all things I think I would like to get done tonight. I think I can do that... So, let's start the party. Apres la party, code development and debugging Friday (for motion), and then Sat. morning

## Guts of the Fish

To take stock (more gut jokes), I'm interested in writing up how I've covered most challenges. In particular, I'd like to outline for myself how I will wrap these into something wonderful, succinct and less-error-prone.

So, the hunk API is easy to set, and fairly similar to the javascript implementations, although a bit messier. The trickiest pickle is (as is tradition) the state-set-change-notifier fn. This is probably because this is the only way we deliver data to hunks outside of their loop / outside of ->get() calls, which are intentional and planned for. Another tricky bit is setting up array lengths (of inputs, and state), bc they aren't nice JS Array classes, they're contiguous hunks of memory in real space.

### Void \*rs, Len, Types

At the heart of the cpp roll is a way for our universe to transport data from an output to an input, without knowing what the particular type of that input (or output) is. This is easy in JS because nothing means anything, but cpp is a land of rules.

The other side of the heart (appropriately), is an interface between these contiguous blocks in memory and the outside world. This is kind of the contiuum of what the managers do... and there's your desire to make this beautifully nesting bytes-types-indices-are-routes-all-the-way-into-the-write move... if we included all of the instructions as well, we could cast all of linear-spaceless computing architecture as a graph also (?) ok.

The other side of the heart, as I was saying, is the interface that pulls those bytes from their blocks in contiguous memory and puts them on the network: into serialized space. This is real talk for Inputs (that receive bytes from network space), States (that are optionally set in the same way), and for Outputs (which deliver bytes to network space, via a wrap performed by the link (same as inputs)).

In order to do each of these things, a blind (void \*rd) manager (and link) must know a few things:
 - the type 'key' (the byte that tells network friends wtf is succeeding)
 - whether / not this is of variable length: alternately, the manager can keep a list of keys-that-need-len-spaces. since we're specifying keys, this knowledge can be a-priori also.
 - the length (in bytes) of the currently-present data. for static-sized types, this is always set to the len of the static type, for dynamic types, is set during calls to op->put(), and sequentially to inputs with ->transport(). These are equivalent to state->set() and whatever wrapper onChange.
 - the actual pointer to the data's 0th byte.
 - a string, also, to name the thing, for humans

### Inputs and Output Wrappers

Inputs are wrappers on data blocks. Because we want to make writing hunks easier, straightforward, and beautiful, we wrap these with a type-specific ->get() function. This does two different things: for static types, it recasts the void* as the type that it is, and returns that to the caller. For non-static types, ```->get(type\* arrHead, size_t\* incByLen)``` it accepts and array of the nonstatic type(d) objects, and writes into that array, by the len incremented (by ptr ref). In either case, it sets the inputs' .io flag to false: the data hath been taken.

### Outputs

Outputs are nearly the same. To ->put(type) we can just ->put(var) for static types, and for variable lens, ```->put(type\* arrHead, size_t len)```. In either case, the call is rejected (by returning false) if the output .io, and if it is !.io, the call moves (by memcpy, probably) from the op to the void* within (by len). It also commits .io to be true, and .wasPosted to false.

Outputs also store lists of connections. Ah connection in the list is just a ptr to an input. **aside:** in another world, or this one, the output could store, in a connection, a global map of the other input: just a 3-byte (min) 16b inputHunkIndice, 8b inputInHunkIndice. the pointer is kind of nice, though, just because a hunk can reorganize its inputs and outputs and doesn't have to commit to all outputs that are connected, new 8b vars... you know what I mean **end aside.** To attach things, we call output->attach(input), or output->detatch(input). Outputs / inputs I typically think of as being lopsided, then, with Outputs being the large bois, inputs being lightweight.

### Transport

So, transport is the part that Inputs or Outputs don't see. Once a loop, the manger runs over each hunk and walks output's lists of connections. There's a pretty simple statemachine that follows our flow-control rules, but tries (in vain) to minimize work.

Once set, outputs are cleared when all downstream inputs are clear, and new posted data has been moved to those inputs, occupying them.

Outputs that aren't connected to anything in the graph have data ghosted into the universe by auto-setting .io back to true.

Pretty sure this is just three bits of output state, depending on the n bits of data in the downstream (n) inputs.

### Network Linking

This is where I can really win in typing stuff. I can probably keep a page (like, a typeset?) of memory structs, each having those things I need: void* (that gets malloc'd on init?), typekey, typestring, and isVarLen: the bit for whether/not we need array type access. Then, for each inputType, outputType, stateType, I can add the same type_t hahaha ... to each one. Maybe some better name... thing->dblock->voidptr, thing->dblock->len etc...

The tricky pickle is probably where size !== len ... for now, could make a rule that we don't malloc more than 1kb per varlen type? Or each varlen type just gets 1kb, and thats it? Each type has a size and a len, sometimes they match?

### Eadianness

I've been through two eadianness bugs. Both systems use little-eadianness (msb first) internally, but 'networks run big-eadianness (msb last)'. To date, I've been trying to stick to big-eadianness, but I also want to be able to memcpy() -> to pull bytes directly off of the network and into memory.

To implement, I'll try first to unf- my prior implementation. I think this is limited to uint16s and uint32s now...

### Real Hardware, Abstract Software

The high level exercise in this work is to develop a computing model that comes to grips with hardware realities: that's true of the network constraints we pulled into our computing model, and it really hits the metal in a firmware application. I.E. Ponyo.

I'm at this impasse at the moment: my micro has only a certain # of resources: at the moment this means interrupt service routines for a timer, but more broadly it is the micro's peripherals. Walking C++ into a world where some resources are singlet and static is troublesome. What's more, most micros have complex (and unique!) methods for multiplexing / turning on / off particular peripherals to peripheral pins. And! things like event systems & configurable logic gates.

Since our environments are self-reporting, we should be able to represent all of this stuff with dataflow. We should also be able to develop an intelligent resource-allocation scheme that would let software abstractions (hunks) request resources. If they are already allocated, we get a no, and the hunk doesn't load, upstream this is a readable error, application building goes on... Better yet, when we ask for 'available hunks', only those whose required resources are available are listed. I.E. network ports drop off of the list as they are added. If one UART is used for SPI on a driver, network ports that use that UART are not listed. Pins, etc.

For now, this is binned as 'future work', although to me this seems like it would be a really wonderful contribution. Self reporting, and system intelligence at the edge (rather than heavy handed software abstractions), with the routing hindbrain, seems like a way into high performance **and heterogeneous** systems.

So: ISRs: at the moment, I keep a global f'n to request ISR hookups...

OK, I tried to hack it. But I can't hack it. I'm going to have to do this properly...

To me, this sounds like making singleton classes for perhipherals. I'm actually kind of stoked to get a prototype of this running, but I think it's going to flex my noodle. I'm not even sure it's going to solve my problem.

The thesis's diamond center: at the edge, we need to represent these relationships. We are trying to re-cast as physical what was physical but was then made (by micro mfg.) software... It's clunky. We can self report, we can embed the datasheet in the environment by way of making allowances that a system developer uses: the tool speaks back. Construction of applications is a dialogue.

Through some struggling, the way to this is invariably the manager. Ponyo is like a living datasheet - it should know what's used, what's available. To prototype some of these processes, what I'll do to handle interrupts first is to make ponyo a singleton, that way I can include a reference to the manager wherever else I might need it.

Once it's singleton'd out, hunks can request interrupt routines from it... Hunk* types will have handles to interruptHandler fns... let's see...

### Current Status

[2019-06-16]

After an arduous battle with JS, I'm (soon) making a return to CPP. In steps, this means: (1st) clarifying the link's role, and it's statemachine-ness. Dumps messages when nc, but tries to open up. Data ports, perhaps also, should dump messages if they're closed (would this mean a dataport-level connected-ness for your phy?), and throw errors? Then (2nd) make sure that managers don't hold back anymore: they are stateless w/r/t connections. That's a hack that you can remove, nice, (3rd) is writing essentially the typeset.js equivalent for cpp, integrated with the link... I won't try to lay it all out here, that's what the work is. Fun fun.

[2019-05-07]

The Ponyo Return

Primarily, with typeset.js, something similar. This is synchronous with writing nets, consider a link that uses those fn's ? Basically just need to key. And we don't have to do like, numchecks etc. Bytes are bytes, bless u cpp.

Also, the manager-inside-of-itself, and the link -> that means adding dynamic hunk updates.

 - first thought is w/r/t the links ... a reasonable bootstrap is going to launch with a default link, cuttlefish will have to open that up and change it as it loads the recursive program ... or, a loaded program will have to update that link as it goes

[2019-04-16]

...

Implemented void* data passing with wrapper classes for put() and get(), so hunk API is mostly set, aside from state.

Implemented void* passing for longer types using virtual transport() f'ns.

I want to start to chat with cuttlefish, nautilus. I think that starts by writing my wrap over the USB/CDC/ArduSerial implementation that delimits bytes via COBS. I want that escape trunk as well, and should copy out .print(vars) functionality. I want Trunk.print() and Trunk.write() or Trunk.write(location, len) to ship via flow-controlled COBS alongside other messages. Wrap in #define.

I've wrapped up Arduino's Print class, which [extends nicely](https://playground.arduino.cc/Code/Printclass/) (thanks Arduino), which will leave me a nice debug back-door (or [trunk](https://en.wikipedia.org/wiki/Escape_trunk)). I wrapped this in a [Singleton class](https://gist.github.com/pazdera/1098119) - which was a nice cpp concept to learn about, that I expect will be useful to hardware. Next up I'm going to push these datas out of a COBSerial port, which will ultimately take something like COBSerial.put(unsigned char* buf, uint16_t len);

Then I'd work towards the missing ```class Link : public Hunk``` implementation, that will dictate how we serialize messages, and how we type-set link inputs / outputs... manager side needs messages to describe changing hunks, then... and hunks need to be able to add inputs to themselves at runtime.

So, link involves drawing that message tree, and making wrestling choices about JSON, or some other more elegant solution.

## Notes June 2019

Ok, back on ah nother pass through ponyo. For the 'trunk', I'll have it write a link-mimicking 'debug' port - I can write an app for this - that dumps ah (tbd) ll-debug msg in front. The link can log that direct, no problems.

OK, working through state. I think this is mostly through, not really any way to check though until we get to adding to that link, so. It compiles. Ok.

Now I need to roll through the link's 'init' / setup ... this is just programming. Once I can start up a link, I can try to hook up thru & thru and receive a manager message from ponyo / cuttlefish. Then reply a brief... etc.

OK. Walking through all of this via link is arduous, and I don't know enough yet. I'm going to make a static link-type first, and see if I can make it operate, and describe the program upstream. Then, see if I can write an led-blinker hunk, describe that, and use a state change to change its internals. That will have been much stuff. So:

 - link does basics for an init having data, mgrmsgs, and numin, numthru
 - boostrap connect data -> cobserial,
 - boostrap connect link mgrmsgs -> manager,

  - ok, those all look fine, will do
 - nautilus ponyo hello cuttlefish
 - link loop code,
 - find message at manager
 - reply with brief
 - reply with hunk definitions

ok, I've got a message down to the link. to pass it on, I need to write the type interface. I have a proposal up to here now, using a typeset.h structure to read and write bytes. pray for me, bc many many void* things going on.

First up, I want to change the way types are written in. States and Nets should use this method ... using typeset, and like TS_UINT32_KEY and TS_UINT32_STRING

OK, going through this, writing readLenBytes / writeLenBytes, not worth effort and bug-proneness. Probably. will treat all strings, indicies, etc, things with lengths, as being base uint16, that's 65k top, that's enough per packet at least. one extra byte in packets is probably faster than all of this kicking aroudn in the cpu. We think. Will have to roll this through js typeset, but it will (?) probably be localized to typeset.js, and a few others?

This puts a serialization-level limit on the size of objects that can traverse links, possibly problematic if later on we want to have links pass big objects by doing link-level packet work. But it's not worth our time at the moment. There's a 1024 memory-width limit on most embedded ports anyways, so here we are.

## Notes May 2019

I'm imagining this is going to cull a lot of the fat out of the other contexts, also, and purify the ideas some... The constraints of microcontrollers, low memory, and the tightness of cpp is welcome at this point. JavaScript is a hot mess and I'm excited to step out of it, hopefully on my return I'll see some elegance sweep back into the js.

So, what does a manager have to do?

Well, first, I have to write a few software objects.

```hunks
Hunks,
 -> have ids, and names
 -> optionally have inputs, outputs, and state
 	-> inputs have names, data types, and state (io, ie)
 		-> they have fn's to .put() data, that the manager uses
 		-> and fn's to .get() data,
 	-> outputs are essentially identical, but the manager .get()s and the hunks .put()
 		-> outputs also have arrays of connections. this is up for debate, as the manager might just keep a list of these connections. it's convenient to hold the array in (or in ref to) the output object ...
 		-> the manager uses handles like .attach(input, pid) .remove(input, pid)
 	-> states have a .value, .type, and .name ...
 		-> managers want hooks in here, to send state updates as messages ... inside of the state is probably the biggest contradiction of the system. maybe we kill it ?
 -> bigtime, they have a loop() function, that the manger calls once a round,

Hunks are objects that extend these classes, and run their loop functions to check input and output states, and if satisfied, do stuff with the data on them.
```

ok, those are rules for objects, managers

```managers
Besides the hunks, the manager has a simple job.
 -> 'transport' data from outputs to inputs, i.e. move memory about
 -> divide time between hunks (call their .loop() functions)

We can think of managers as the tiny universe in which hunks live. In the universe, data moves from outputs to inputs, and time cah-chunks forwards one 'loop' at a time.
```

What about recursion? Representation?

```programs
To store a program, we keep a simple list of hunks, and a list of links.

(?) programs can define inputs and outputs, so that we can nest them as hunks. so, programs are big trees / graphs of hunks. loading and saving programs walks this tree.

(?) programs are nested in their description and representation, but they are not managers... the manager (a thread) walks these trees to dish links and to run loops.
```

More about that manager ...

```managers
Managers are Hunks, Too. They run a loop (within which they run their hunks' loop). The can receive inputs and write outputs... This doesn't have a lot to do with program execution, but it is the mechanism by which we render contexts and defines how we edit, load, and save programs... their (messages) inputs and outputs.
```

So far we have described hunks' operation, how they pass messages, and how we can describe programs and nest them. What about multi-threading and running programs across a network? For this world, we have a few tools.

```links
Links are Hunks (just blocks of code) that ferry messages from one context to another. With links, we can take serialized data from other contexts, and pass them along the context's local dataflow paths. Links do the opposite job, as well, of taking local dataflow inputs and serializing them for remote objects. They bring flow-control and add a routing layer to any data link that we can ferry data across: UARTS, SPI, Ethernet, TCP/IP, 802.11, Bluetooth, etc. Data Links / Drivers are just another Hunk, so we can swap them out for whatever physical line that we want.
```

Simple enough... Sounds like we can compose programs across multiple threads (physically separate or not) ... but how do we compose them, see what they're doing? This is cuttlefish ... and it's assumed that humans will do it. Then we do views -> routed through links -> managers. Rather than taking networking for granted, contexts have to bootstrap themselves with a program that includes a data driver, a link, and its connections to the local manager. Ok.

## Types, Serialization, Memory Allocation

The exercise now is into the gritty details of memory location, movement etc. And, again, implementation of dataflow-type worlds inside of linear-space computers.

See Protobuf, and JSON. JSON is great, but heavy (because we carry keys around) and Protobuf is great, but cumbersome (because we have to cross-compile).

Programmatically I'm pretty sure how I'm going to do this. Links (perhaps I should rename 'links' hunks to 'bundles') are the spaces (in memory) that outputs pass-to and that inputs pull-from. With CPP, this means that we can have Output* and Input* types that, when we ->put() and ->get() from them, are accessing a Link_uint32_t*, or something similar. The Link_uint32_t is where the memory is genuinely located.

However, I have to have put(typed) and get(typed). So I should perhaps just have these as classes / templates.

Big problems for this are...

Types: some std list, and then arrays of those stds ... an array like
 type : numof : [data ...]

### Jake Learns about CPP

Running embedded systems on CPP is a bit of a hot take if my reading of the internet blogs is correct. However, it's also becoming more and more popular, especially as people work towards more complicated firmwares. Certainly folks who are writing embedded OS's (RTOS etc) are all out there in CPP.

This is slightly contentious because CPP's object oriented structure does not exactly reconcile with the truth of embedded systems: i.e. that there are typically only one (1) of each 'object' - i.e. there may be 5 UART ports, but only one is USARTC1, if you know what I mean. In embedded computing, time and space are real, and we have to consider that while we write software. In CPP, we can ostensibly have *thousands* of the same 'object' ... so, here we are.

Here's [a blog post that I'm reading](https://bitbashing.io/embedded-cpp.html) written by wizards.

I'm thinking this is actually an interesting application for DDMC ideas about 'conversational programming' - i.e. requests to load certain resources can be rejected by Ponyo given their current use in other contexts. I.E. I can keep a list of hardware resources, and 'check them out' on loads of modules that use them, and I can check use when I go to load another module.

Ok, so, let's see. Since I'm not planning on nesting any managers, I can pretty much write the thing headless. I need an array of objects.

I need a good test case, that will stretch me through a few of these problems. (1) I'll write a hunk for the UART port ... this can be my data link. (2) I'll write this keeppalive hunk, that blinks the clock light. (3) I'll write a tiny manager that loads them both up, and rolls over their loops.

OK ... this is OK, but I should be bootstrapping like 'loadProgram(object)' or something like that. *or* oh wait, we're going to do that fancy shit remotely. I need like 'loadHunk(name)'

Beginning to wonder if the string library ... arduino ... *is* tempting. Specifically, all of arduino's built in String() functionality might a big ol' boost to get going. And my is that terminal nice.

#### PTR Incrementing / Dereferencing,

**increments the value at the location of the pointer:**
```dest[(*dptr) ++] = TS_UINT16_KEY;```

**increments the pointer itself**
```dest[*dptr ++] = TS_UINT16_KEY;```

This from [this stackoverflow q](https://stackoverflow.com/questions/859770/post-increment-on-a-dereferenced-pointer), and worth time for all of this serialization ... writing into / out of buffers quickly.

### Notes from Erik

```cpp
class Hunk {
public:
    // Need the destructor to be virtual so that when you destruct a Hunk*
    // that's actually a KalmanFilter, the KalmanFilter destructor gets
    // called.
    virtual ~Hunk() {}
    virtual int n_inputs() const = 0;
    std::string name() const { return "Hunk"; }
};


class KalmanFilter : Hunk {
    int n_inputs() const { return 2; }
    std::string name() const { return "KalmanFilter"; }
};

sizeof(KalmanFilter)

int main() {
    int 5;

    // stack
    KalmanFilter filter;
    filter.n_inputs();

    // heap
    KalmanFilter* filter_ptr = nullptr;
    // new allocates the memory and calls the constructor
    filter_ptr = new KalmanFilter();
    filter_ptr->n_inputs();
    hunk_ptr->name(); // returns "KalmanFilter"
    // delete calls the destructor and frees the memory
    delete hunk_ptr;

    // heap
    Hunk* hunk_ptr = nullptr;
    hunk_ptr = new KalmanFilter();
    hunk_ptr->n_inputs();
    hunk_ptr->name(); // returns "Hunk"


    // this only allocates memory
    void* data = malloc(sizeof(KalmanFilter));
    // this only calls the constructor
    Hunk* another_hunk = new(data) KalmanFilter();
    // I think this only calls destructor? Don't do this.
    delete another_hunk;
    free(data)
}

class HunkPool {
public:
    Hunk* make_new_hunk(std::string hunk_type) {
        switch(hunk_type) {
        case KalmanFilter:
            hunk_pointers[n_hunks++] = new KalmanFilter();
            break;
        default:
            // error
    }

private:
    int n_hunks;

    // new way
    std::array<Hunk*, 256> hunk_pointers;
    //old way
    Hunk*[256] hunk_pointers;
};
```

### Nets: Manager Accessible F'ns and Internal Fn's

I need to write software objects for Outputs and Inputs. Some of these will pass uint32_t, some floats, some doubles, etc. These should be easy to type, and will be specified by name, and serialized upstream.

What's problematic is that I'll need manager-global handles on these things, to operate them.

Nets (outputs and inputs) are going to have to be typed. Things that are certainly typed (have to be sub-classes) are:
 - the 'hunk api' level:
  - put / get
 - the output has to transport to the inputs
 - as a result, the output's connections[] list has to be a ptr of input_typed;
 - i *think* the result of that is that output::attach(input) has to take string ref ... i.e. we can't pick through the manager to find by typed ref.

This seems a bit problematic. Hunks have to keep lists of outputs and inputs. So we somehow have to attach them via their generic types...

One way is to write attach au manuel, and then have the *output* search through some registered list of all inputs of the same type ... this seems kinda bonkers. I couldn't even have it just search through the list of hunks, bc it would have to ptr->thru->generic->inputptrs;

I can maybe just attach input* to inp_uint32* and see what happens? I am guessing that the compiler is going to yell at me.

*ok* generic inputs contain something like data_len and data_ptr, and they malloc() some space ... the copy does a data pass by memcpy(len) ...

OK, going to go forwards with this. Also going to mash nets together ... This worked out well, I implement generic void* data storage inside of nets (inputs and outputs) and move data between them with a virtual transport() function, that I can extend for unique types, i.e. adding length values to put() and get().

#### Naming and Typing

(1) naming / typing (probably input-output classes per type, then don't have to name type au manuel)

I need to write software objects for Outputs and Inputs. Some of these will pass uint32_t, some floats, some doubles, etc. These should be easy to type, and will be specified by name, and serialized upstream.

Oh wait this is easy, I do it with class Inp_uint32_t : Input {

}

Just a bit unfortunate that I have to write out every goddang input and output, but hey haha...

Here's the generic Hunk that a manager sees (as of April 16 2019).

```c++
#include "transports/nets.h"

class Hunk{
private:
  String id_;

public:
  virtual ~Hunk() {};
  String name;
  // naming, getting name
  String getId(void) { return id_; }
  boolean setId(String id) {
    id_ = id;
    return true;
  }
  // manager calls these
  virtual void init(void) = 0;
  virtual void loop(void) = 0;
  // everyone's got (pointers to) some
  Output* outputs[16];
  uint16_t numOutputs = 0;
  // inputs
  Input* inputs[16];
  uint16_t numInputs = 0;
  // no state yet
};
```

Hunks extend this class, I don't have a great example of a 'typical' hunk at the moment. But those inputs and outupts are both generic 'nets' ->

```c++
#define MAX_TRANSPORT_MULTIPLEX 16

class Input{
public:
  // ref for sys and humans
  String name;
  String type;
  // size in bytes
  size_t size;
  void* dataPtr;
  // state
  boolean io = false;
  // actions
  // pls write 'type get()' as an interface to those bytes
};

class Output{
public:
  // naming
  String name;
  String type;
  // tha beef
  size_t size;
  void* dataPtr;
  // state
  boolean io = false;
  boolean wp = false;
  // the business
  // built in to output
  boolean clear(void);
  boolean isClearDownstream(void);
  Input* connections[MAX_TRANSPORT_MULTIPLEX];
  boolean attach(Input* ip);
  boolean transport(void);
  uint16_t numConnections = 0;
  // *does* have to have connections ...
  // pls write the interface; boolean put(type);
};
```

Inputs and outputs do work, here's the .cpp

```c++
#include "nets.h"

// 'clears' just by setting flag,
// ... these are state machines
boolean Output::clear(void){
  io = false;
  return true;
}

boolean Output::isClearDownstream(void){
  // check over inputs, return
  boolean clear = true;
  for(uint16_t c = 0; c < numConnections; c++){
    if(connections[c]->io){
      clear = false;
      continue;
    }
  }
  return clear;
}

// we can actually do a lot of this non-specifically
boolean Output::attach(Input* ip){
  if(numConnections < MAX_TRANSPORT_MULTIPLEX){
    // we can typecheck at attach
    if(ip->type != type && ip->size != size){
      return false;
    }
    // ok then,
    connections[numConnections] = ip;
    numConnections ++;
    return true;
  } else {
    return false;
  }
}

boolean Output::transport(void){
  // we r clear 2 go, here we will move some data
  // grip it
  for(uint16_t c = 0; c < numConnections; c++){
    // and rip it
    memcpy(connections[c]->dataPtr, dataPtr, size);
    // these are occupied now
    connections[c]->io = true;
  }
  // data hath been posted
  wp = true;
  return true;
}
```

I extend these nets by-type, so that I can define unique put(dataType) and dataType get(void) functions;

Here's a uint32_t typed net ->

```c++
#include "nets.h"

// INPUT
class Inp_uint32 : public Input{
public:
  Inp_uint32(String nm);
  // actions
  uint32_t get(void);
};

// OUTPUT
class Outp_uint32 : public Output{
public:
  Outp_uint32(String nm);
  // actions
  boolean put(uint32_t);
};
```

So far, I like this - although I'm definitely feeling like a CPP beginner, not fully expressive with it yet. Seems like things can get cumbersome, and there's ample room for yak-shaving.

I know that I will want to define and pass along variable length arrays of data, particularly in the case of passing messages from a link to different PHYs, that's something I'll want to do almost right away. So I think that's about my next task for today: pushing strings about. I think I can do this via a similar mechanism, but it might just take some more articulation. This will also give me another chance to write another typed net, and write an example program that does stuff with that serialport. OK. Next step is to imagine that program, write it.

Then I'll be interested in hooking up to cuttlefish / nautilus, working through startup procedure practices (for the full chain, relinking, making my own environment my debug baby) etc. Focus, !

#### Serialization

Once I have nicely typed inputs / outputs, I think I should circle around to writing my baremin manager that communicates to cuttlefish / nautilus. Then I should begin prototyping systems that live on all three, all the time. Big deep dev dev. Eye of tiger. Get it.

The first thing (i'll guess) that I'll need is the packet interface / serialization across links solution. This is going to require the simultaneous convergence from links across the isle.

OK, I've COBS encoding on the way out, now I need in on the way in. I'm going to wipe my old serialport hunk and write COBSerial, this assumes using Arduino's builtin Serial class to listen for bytes (which is an adafruit usb cdc implementation, nice and snappy). This should be a straightforward thing to adapt to buffered / DMA'd UART instances. I after wrapping with cax.js (soon) I can take a day with Nautilus / Cuttlefish to see if they can become tenable development environments for me... seeing what will make them stable across booting / rebooting / auto-listening / auto-opening ports in various state etc. My first message type will be the debug, delimited by 254 to start, assuming the entire length of the rest of the packet is to be printed out as a string.

So, I'll have delimited packets. Because the network is where things reconcile themselves, I'll have to explicitly type anything I want to send between contexts. So, types at link-inputs will have to belong to this bounded set, there's just not really any other way around that.

Packets will land at links de-COB'd, as byte arrays of a known length. My proposal for typing data within the packet is like this: packets are lists of data objects:
```
typeKey,len(?),b0,b1,...,blen : typeKey,len,b0,b1,...,blen : etc ...
```
for speed, I propose to keep some typeKeys as len-less; i.e. everyone can know that uint32_t's are 4 bytes long.
*caveat* is that using one len byte restricts our types to 255 len objects, i.e. this maxes out our string type. That's approximately one tweet...

This is going to be a bit tricky and I will probably want to test-by-using it. In the immediate term, I know I can cover my bases with just a few sets:
 - singletons and arrays of the basics: chars (arrays: strings), 32integers, 32floats ... it might be a minute before I need more than that.
 - I *want* a heirarchichal description of types, where I as-it-goes reduce a char array to types
 - to start, link ports will faithfully pass packets onto ports, ports being the # in the 0th byte ... they will type-check as they do, and as they pull in, for the #'s type.
 - the 254th 'port' is el debug channel,
 - There's a decision to be made about link- messages. I am actually feeling like 0th port should be available to send requests to the link to add links of certain types ... but this is also a capability we want to be able to build into other hunks, there was an earlier proposition to do it via some state objects and messages from the manager
 - here's the other option: To keep things nice n dirty, I will route messages out of the link: the link won't be responsible itself for handling any string-type messages. To, say, add more outputs to the link, we can do this via the manager. Does that make sense? I will have to type those outputs though...

### HERE IS WHERE YOU ARE AT

 -> moved some names around,
 -> completing a test of state updates requires us to send a message down to ask for a state update from UI, so we should hookup, write the manager as a hunk with inputs (or global object) and write that tree of messages (interpreting from strings I mean) this will also involve casting unsigned chars as Strings to == them from the message ... a good chance to ask erik about allocation, and garbage collection

To go forward, I think that I need to solve those link-change-etc messages between cuttlefish and nautilus. I understand the requirements that cpp and genuinely memory-respecting languages need me to make now, so after I finish with COBS on cax.js and ship something back and forth, I'll revisit cuttlefish and nautilus to serialize as described.

For compartmentalizing, the ```Link``` class will assume un-cobs'd packets, and will assume that the datalayer will carry out flowcontrol for the link. I can write a COBSerialPFC ... COBSerial Packet Flow-Control, and rely on shipping that bytes, flowcontrolling via manager. OK. That's to come.

Ok. handles. After that is writing out a bunch of hunks, probably lots of program load, reconnect attempts, init state decisions etc.

 ... -> back to nautilus, cuttlefish.

At first I was imagining that my main serialization task would be describing program data types... addHunk, addLink, etc. But this is easy: a few keys, and string matching. Or, manager messages are all lists of strings, the 1st being the type of message it is, the latter being the arguments that are passed into that respective f'n. That ain't no thing. Wants a heirarhichal type declaration of like, 'strings,len,string1,len,b1...bn,...,stringn,len,b1...bn' though. These can be 'msgs', perhaps? Human readable trees, kindof.

There's this idea of keys-by-call-and-reply...

And I think perhaps that's what I need, not really for the manager messages (although used there as well) but for also the hunk objects. I.E. these things are going to have particular data structures to their inputs -> ultimately, some array of bytes. In CPP this is obvious, in JS it is obfuscated by already what JS does, keys->lookups. We need a way to serialize not just messages to the context, but messages within the context as well.

How do I do this without taking on all of cpp, templates, etc? What are tests I can run, to start?

Here we are: April 17 2019, working on implementing some COBS as a first step towards serialization. While the abstraction purist would write COBS as abstract from a particular implementation... ok I will do this.

So, COBS is a flow-controlled object in the graph. To start, because I want blocking / reliable trunk prints, I can expose a function to do so. Its inputs will be charArray types, eventually written by a link type.

Like, this is a crutch. I shouldn't be afraid to break things. No more serial print except for via cobs: rules. Debug messages will be the first 'type' you print. Maybe I need to learn about operator overloading?

OK:
 - writing COBS on the cax side, testing reversal, finding that 0, 1-> bug ?
 - very close, nearly there, many things coming
 - to wake up,
  - manager is a hunk in a bootloop, or ?
  - the link
  - serialization tests
  - schedule yourself for board arrival thursday, be systems ready to just be fleshing out hunks by then ... one day with js / div drawing ?
  - program load / etc routines ? on state ?

#### State Objects

After tackling links, with void*rs, State objects will probably resolve similarly.

OK, basically, I need to write a hunk-and-state-object pair-specific handler function that the manager can call, probably it will call it with the contents of a message just received. So, like, (inside of manager loop, after message)

```c++
if(hnk->states[s]->change(void* dataPtr, size_t len)){
  // returned OK, meaning we can update upstream
} else {
  // easy-return error message
}
```

this would be nice because then I can
 - pass the dataPtr straight from the message's memory location, and type-and-check in one place