lottia notes

Fortune habits in Nix

2025-07-26T05:10:00Z

I recently saw and liked Judy2k’s “Using fortune to reinforce habits” — building effective habits is an under-appreciated part of learning any discipline. (And xh is one I had the exact same trouble with.) Let’s implement it in our Nix setup.

Step 0: don’t install `fortune`

You don’t have to do this. That’s probably one of the neatest parts of Nix right there — you can have fortune embedded in your daily workflow without having to expose it globally, without accidentally tab-completing strfile or rot when working and having no idea why that’s a thing.

Step 1: write your habits file

This file goes into your Nix configuration, whether it’s a nix-darwin flake or a vanilla NixOS install with all your config in /etc/nixos.

Write your lines, with % separating entries. Don’t run strfile on it — remember, we didn’t put fortune in our PATH, and we don’t need to do that work, nor do we want to add a manual step (running strfile) for us to forget later.

I’m assuming it’s called habits hereon. Here’s what my first version of habits looks like:

mtr for traceroute.
%
entr to re-run on file change.
%
xh is our HTTPie replacement!
%
fd [pattern] [path ...]
%
Try using gg for jj.

Start small. Once you’re seeing these regularly, you’ll also remember where to add new ones when they occur to you!

Step 2: write a derivation that compiles your habits file

We want to run fortune’s strfile on the habits file. We need to store the result side-by-side with the input; the output of strfile is just an index into the original.

let
  inherit (pkgs) fortune;
in
pkgs.stdenv.mkDerivation {
  name = "vyx-habits";
  src = ./habits;

  unpackPhase = ''
    true
  '';

  buildInputs = [ fortune ];

  buildPhase = ''
    strfile $src habits.dat
  '';

  installPhase = ''
    mkdir -p $out
    cp $src $out/habits
    cp habits.dat $out/habits.dat
  '';

  passthru = {
    inherit fortune;
  };
};

I put this in config evaluated by Home Manager, but you can put it anywhere you have access to Nixpkgs. Let’s go through it step-by-step.

First, we bring fortune in scope. The package in Nixpkgs has fortune and strfile in its bin output.

Now we create a derivation. We include the habits file we wrote as the src attribute, and replace the unpackPhase with a no-op since there’s nothing to extract.

fortune is the lone build input, meaning that its bin outputs (including strfile) will be in $PATH. To build, we run strfile on the source. The source will be in the Nix store, so we supply an output filename — by default, strfile will try to write its output next to the input.

To install, we create the output directory (this is a Nix-ism you’ll see everywhere), and then copy both the input file and the compiled index there.

Finally, we pass through the fortune package we used as an attribute on this derivation itself. We’ll use this to actually call fortune later.

Lovely! We now have a derivation that indexes our habits file, and puts the result somewhere that we can pass to fortune and have it Just Work(tm).

Step 3: integrate with your shell

This will vary depending on your shell setup. I’ll show how it works with mine.

I configure fish using Home Manager. One of the configuration options is programs.fish.interactiveShellInit, which is shell code called when an interactive shell is initialising. I set the fish_greeting variable here, which the default fish_greeting function just displays.

I bind the above derivation to the name habits, and then set fish_greeting by invoking the passed-through fortune package, with the argument being the base name of the complied habits. (fortune will look for the .dat file next to it.)

let habits =
  let
    inherit (pkgs) fortune;
  in
  pkgs.stdenv.mkDerivation {
    # ... (unchanged from above) ...
  };
in
{
  programs.fish = {
    interactiveShellInit = ''
      # ... (elided) ...
      set fish_greeting 'Nyonk! '(${habits.fortune}/bin/fortune ${habits}/habits)
    '';
  };
}

Note how our use of the passthru attribute means we don’t actually have the fortune package in scope anywhere except building the derivation itself; this is a nice kind of clean, and means there’s no chance of e.g. using a different fortune to compile the index than what we use to read it. (There’s probably very little chance of a breaking change here between fortune versions, lol, but imagine (much) bigger systems and you can see how this could be handy. :))

Step 4: prophet

Yay, we’re done!

Now, whenever you modify the content of habits in your configuration, a new derivation will be built, and your shell init will use it.

Let’s have a look at what our built fish config looks like:

$ grep Nyonk ~/.config/fish/config.fish
set fish_greeting 'Nyonk! '(/nix/store/543q6d77f4p27572xb9c5wngg7mg5rh8-fortune-mod-3.24.0/bin/fortune /nix/store/ggp28h3cmvciyaxfr0rxaz154ljap89l-vyx-habits/habits)

oh yeah i love horizontal scrolling. And what does our derivation’s output directory look like?

$ ls -l /nix/store/ggp28h3cmvciyaxfr0rxaz154ljap89l-vyx-habits
total 8
-r--r--r-- 1 root wheel 134 Jan  1  1970 habits
-r--r--r-- 1 root wheel  48 Jan  1  1970 habits.dat

Neat!

Step 5: next steps

You could try some of these:

Instead of just packaging the habits, write out a script that actually calls fortune with the supplied data. Then using it is just a matter of invoking your package’s bin output.
Alternatively, put together the calling syntax in a passthru attribute, and just interpolate that into your shell init. Shell-specific, but kinda neat.
Want some experience writing NixOS modules? You could write a module which builds this and injects it into your shell config with a single enable = true;. Bonus points for accepting the habits data as a configuration option!

TextMode Designer

2025-01-06T01:58:00Z

Here’s a video of the TextMode Designer in use, fixing the control order of a dialog used in the ADC, and one of designing a new dialog.

Comptime, flow, and exhaustiveness

2024-12-17T10:02:36Z

Busy working on the ADC lately, but I just happened upon this kind-of-follow-up to the non-intrusive vtable.

I have methods implemented by only some “subtypes”, and usually “do nothing” is the correct default, or perhaps “return null”. They end up looking like this:

pub fn handleMouseDown(
    self: Control,
    b: SDL.MouseButton,
    clicks: u8,
    cm: bool,
) Allocator.Error!?Control {
    switch (self) {
        inline else => |c| if (@hasDecl(@TypeOf(c.*), "handleMouseDown")) {
            return c.handleMouseDown(b, clicks, cm);
        },
    }
    return null;
}

fn handleMouseDrag(self: Control, b: SDL.MouseButton) !void {
    switch (self) {
        inline else => |c| if (@hasDecl(@TypeOf(c.*), "handleMouseDrag")) {
            try c.handleMouseDrag(b);
        },
    }
}

’tis a fine barn, but sure ’tis no pool.

What I would keep encountering was that I’d write an implementation for a “““““subtype”””””, and then run the program and wonder why the behaviour seemed unchanged. It’s amazing how many times you can encounter the exact same problem — in this case, a missing pub qualifier on the implementations, meaning they’re invisible to other files.

What if we made the default behaviour opt-in, instead of implicit? Here’s an example:

fn parent(self: Control) ?Control {
    switch (self) {
        inline else => |c| if (@hasDecl(@TypeOf(c.*), "parent")) {
            return c.parent();
        } else if (@hasField(@TypeOf(c.*), "orphan") and c.orphan) {
            return null;
        },
    }
}

We check if there’s a parent decl, and call it if so. If not, we check for an orphan field, and if it exists and is true, do our default action. Note that we don’t assert this as the only other alternative. Let’s see what happens if we compile an existing control that doesn’t supply either:

src/Imtui.zig:63:30: error: function with non-void return type '?Imtui.Control' implicitly returns
    fn parent(self: Control) ?Control {
                             ^~~~~~~~
src/Imtui.zig:71:5: note: control flow reaches end of body here
    }
    ^
referenced by:
    focus__anon_8053: src/Imtui.zig:352:35
    accelerate: src/controls/DialogButton.zig:67:29

The function implicitly returns! Both conditions evaluate to false at comptime, so the body of the method ends up being totally empty. (Alternatively, if you use return switch, you’ll see a message about error: expected type 'whatever', found 'void'.)

It’s not very helpful, because the reference trace refers to the point at which this function gets called. We don’t actually know which is the missing implementation, just that it exists.

But that’s okay, we can add that ourselves!

fn parent(self: Control) ?Control {
    switch (self) {
        inline else => |c| if (@hasDecl(@TypeOf(c.*), "parent")) {
            return c.parent();
        } else if (@hasField(@TypeOf(c.*), "orphan") and c.orphan) {
            return null;
        } else {
            @compileError(@typeName(@TypeOf(c.*)) ++ " doesn't implement parent or set orphan");
        },
    }
}

Ja nii:

src/Imtui.zig:70:17: error: controls.Dialog.Impl doesn't implement parent or set orphan
                @compileError(@typeName(@TypeOf(c.*)) ++ " doesn't implement parent or set orphan");
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
referenced by:
    focus__anon_8053: src/Imtui.zig:354:35
    accelerate: src/controls/DialogButton.zig:67:29

Finally, note how a field used this way must be: comptime!

comptime orphan: bool = true,

If not, its value won’t be available at comptime, and codegen will need to produce a runtime condition for the c.orphan check, meaning the possibility of false is always entertained and the @compileError will fire.

Things to consider:

It might be worth asserting in the first branch that orphan isn’t set to true, to avoid any confusion about behaviour when both are set.
We only got the exhaustiveness thing because this example returns a value. With void returns, the @compileError isn’t optional if you want to know if you forgot.
Did you buy the graphite tube?
Try comptime opaque: void to avoid the need for @hasDecl() and the boolean check! Does it work? Almost like little tags, attributes, hmmmmmm.

Platform-specific hacks for high-DPI

2024-11-24T11:06:00Z

This was written up while working on Ava BASIC’s IDE, the Amateur Development Client (ADC).

macOS

We set .allow_high_dpi = true when creating the window with SDL, which does the right thing on macOS. We classify this situation as “native hidpi”; we recognise it by noting the created renderer has an output size 2x that of the window in both dimenisons.
- We only need set the renderer scale factor to 2; everything else behaves as if it’s at 1x automagically.
SDL_GetDisplayDPI always returns the native DPI for the display.
- On my Macbook’s 14” Retina screen @ 3024 x 1964, this is around 255x255.
- On a 32” 4K monitor, this is 137x137.
- This appears reconstituted from other figures as it varies with floating-point tendencies between resolutions, but it’s about right.

Windows

SDL’s .allow_high_dpi = true doesn’t do anything.
We use the SetProcessDPIAware Win32 call so we don’t get affected by Windows’ automatic UI scaling.
SDL_GetDisplayDPI seems to give 96 * ui_scaling_factor, so we get 144x144 with the default 150% UI scaling on piret, and not 125x125 like we actually have, but it’s good enough.
We then apply “manual hidpi” (see Linux).

Linux

SDL’s .allow_high_dpi = true doesn’t do anything.
SDL_GetDisplayDPI always returns the native resolution DPI for the display, on both X and Wayland.

Note: Trying to get the acutal DPI/scaling factor in use seems Really Hard™. I actually didn’t even bother trying at all at first, and jumped straight to the current solution, but after I ironed out all three I thought, “wouldn’t it be nice?” Anyway: hahahahahahahahaha. I’m sure there’s a nice way to do it. (hahahahahahaha.)

At some point I started feeling really committed to writing this note, so let’s actually research this properly. The first obvious bifurcation is X vs Wayland. Going any lower than that is a path to madness (i.e. code specific to e.g. KDE, your particular Wayland compositor, etc.) and frankly my life is full of those as it is.

X

Let’s try to get an answer for X first. I normally use Wayland — on booting my machine into X instead (still Plasma 6), I’m greeted with my UI entirely unscaled! I’m surprised. Let’s investigate ways to obtain a figure here:

xrdb -query doesn’t have an Xft.dpi entry.
xrandr gives 2560x1600, and the monitor’s size as 345mm x 215mm (which is correct, and gives us a PPI of 188x189). It doesn’t hazard an attempt at giving any UI scaling factor or DPI setting (despite accepting one with xrandr --dpi? what must that actually do?).
xdpyinfo (nixpkgs#xorg.xdpyinfo) gives 2560x1600 at 96 dpi, and notes the screen dimensions as 677mm x 423mm, which appears to be calculated simply from the previous figures.
Some folks have suggested grepping X’s logs! Where found, it said 96. (Can you imagine if this just had The Answer? “Please specify path to your X server’s logfile or supply systemd unit name to continue.”)

I reset the UI scaling factor (to 150%), and then restarted X (without which about half the items had it applied, half not).

Only xrdb -query’s output changed: Xft.dpi: 144 has appeared. This looks useful — it looks like the 96 * ui_scaling_factor thing as well here. Let’s sigh and verify by checking 125% (it writes, having already gone back into Wayland … ugh. The worst part is the keyboard (I typo my password on it far too often; not used to the Framework), and a bug somewhere between KDE, SDDM and the laptop’s fingerprint reader means logins take about 30 seconds to process, and honestly I’m just one cat up against the world here). Yes! Xft.dpi: 120!

Overall, there’s not a lot; it is clear that most desktop environment toolkits implement this themselves, and so there’s not necessarily a straight answer. Querying the font DPI X resource seems likely to give a useful number, though, when present, and I think we can call that Good Enough™. We have to shell out, which is ugly as hell, or query the X server ourselves.

Wayland

First up, let’s try all the X methods against Xwayland, just in case we get an easy win. This is at 150%.

xrdb -query gives Xft.dpi: 144.
xrandr gives the same as real X. (Not the actual same; a lot is different thanks to the virtual server. But the relevant stuff is identical.)
xdpyinfo does likewise.
There’s a lot less in the logs/journals for Wayland that I’ve found.

What about 125%? xrdb -query gives Xft.dpi: 120, rest the same. 100%? Xft.dpi: 96, the first time we see this result explicitly here.

And it turns out, that’s it: that’s the solution. We can get some minimally useful information for both display managers with the one method.

… but if we did want more information from Wayland, does its design mean we can get it? Turns out, yes.

wayland-info (nixpkgs#wayland-utils) gives us:

scaling factor of 2 in wl_output interface properties; and
logical display size of 1707x1067 from zxdg_output_manager_v1 interface, but maybe that’s not (as) reliable.

The 2 there is an interesting one. It’s 1 at 100%, but 2 above that. Above 200% or so it seems to go to 3. This is wayland_server::protocol::wl_output::WlOutput::scale; I guess it (? Plasma?) gets us 150% by rendering at 200%, and then bitmap-scaling the result? That doesn’t sound right. Oh boy.

The zxdg_output_manager_v1 figure gives us exactly what we want, in zxdg_output_v1::logical_size, but it looks comparatively new/ unstable and isn’t a core part of the protocol, so I assume it’s comparatively unreliable. (Using that we can calculate the actual set DPI, i.e. 125x125.)

I guess on Wayland (with Xwayland) the perfect priority would be zxdg_output_v1::logical_size > xrdb -query > wl_output::scale. But I’m going to assume Xwayland is in fact ubiquitous, and therefore xrdb -query will have to do for both.

(If I actually end up implementing this, I’m probably going to eventually succumb and implement the X calls to avoid the subprocess.)

Okay, but what is the actual solution?!

When the reported DPI is greater than or equal to 100 in either dimension, set the renderer scale to 2, double both the window dimensions, and set the effective scale to 2. The effective scale is divided from cursor positions in mouse events before they’re handled.

wtf

We check this immediately after doing the macOS-style hidpi check, which skips the rest of this if taken, i.e. if the window is on a Retina screen.

Because of that, and because we call SetProcessDPIAware on Windows at startup, “reported DPI” has the following meanings (with examples from above scenarios given):

macOS: native, unscaled DPI. (255x255 on Retina, 137x137 on 4K)
Windows: 96 * ui_scaling_factor. (144x144)
Linux: native, unscaled DPI. (188x189)

So in any case, we see our DPI >= 100 and just run everything at 2x. If the Windows user happened to be running at 100%, then we would too, and would fit right in. If we REALLY cared about that happening on Linux too, we could go the xrdb -query path. That way lies misery, I know it.

Git and jujutsu: in miniature

2024-11-09T03:30:00Z

Last night in bed, I realised we’d encountered a scenario at work during the day where something happened so fluidly in jujutsu that it’d make a good case story! Let’s compare, step by step, how it’d look with git.

The stage is set: you’re working on a big, old, legacy codebase, and you’re 10 commits deep in a branch where you’re adding a new parsing component which will, by the time the branch is merge-ready, completely supplant an old one and all its uses.

The parser is mostly called through a centralised place, which is well-covered by tests, so you can feel reasonably assured that green CI will mean you’re on the right track, and you’ve been removing parts of the old parser as you introduce the new.

But what’s this? There’s an outlying case where a method on some random model calls into the component directly — unlike most uses, it’s just calling the parser to clean up some input. And while the method is covered by a number of tests, this particular function of it isn’t at all — you could replace the parser with the identity function and these tests would be fine with that.

So we need a new test. We’re currently in some other WIP on this branch, and the original parser is half-taken to bits, so we’ll write the tests against trunk, called develop here.

$ jj new develop -m 'content_spec: assert body_excerpt strips tokens.'
Working copy now at: rltuvkoz 9b9f6db4 (empty) content_spec: assert body_excerpt strips tokens.
Parent commit      : xkowykqr 83ad162d develop | (empty) Merge pull request #1736 from backfill-pr
Added 10 files, modified 46 files, removed 4 files

$ git checkout develop
error: Your local changes to the following files would be overwritten by checkout:
        app/lib/rml_parser.rb
        spec/lib/rml_parser_spec.rb
Please commit your changes or stash them before you switch branches.
Aborting
$ git stash
Saved working directory and index state WIP on (no branch): 478c7377 remove local artefacts.
$ git checkout develop
Previous HEAD position was 478c7377 remove local artefacts.
Switched to branch 'develop'
Your branch is up to date with 'origin/develop'.

Now, a seasoned git developer (with a good vcs prompt for their shell) will know to stash reflexively, and indeed the above three commands under my aliases would be co develop, st, co develop, but it’s interesting that we kind of have to context-switch for a moment here and think, “OK, working copy changes, maybe some tracked, some untracked, put them all into this ‘stash’ thing over there so we can move around freely”.

So we write our test, confirm it’s actually asserting the behaviour. We’re now here:

$ jj status
Working copy changes:
M spec/models/content_spec.rb
Working copy : rltuvkoz e0186732 content_spec: assert body_excerpt strips tokens.
Parent commit: xkowykqr 83ad162d develop | (empty) Merge pull request #1736 from backfill-pr

$ git status -sb
## develop...origin/develop
 M spec/models/content_spec.rb

The manoeuvre: jj

Great! We want to introduce this change into our branch, so we can be sure we don’t break this use case. We don’t really want it right at the tip, since there’s a progression of commits; we want it a few commits before.

@  rltuvkoz ashe@kivikakk.ee 2024-11-09 11:55:13 e0186732
│  content_spec: assert body_excerpt strips tokens.
◆    xkowykqr redacted@redacted.com 2024-11-08 15:30:54 develop git_head() 83ad162d
├─╮  (empty) Merge pull request #1736 from backfill-pr
│ │
│ ~
│
~  (elided revisions)
│ ○  xnsrwqok ashe@kivikakk.ee 2024-11-09 11:54:38 ec3df6f8
│ │  (no description set)
│ ○  rvumupsn ashe@kivikakk.ee 2024-11-09 11:54:38 rml-parser* 478c7377
│ │  remove local artefacts.
│ ○  ppmltptr ashe@kivikakk.ee 2024-11-09 11:54:38 25b122ab
│ │  Token: migrate uses of #token to #original_token.
│ ○  pytwvkmx ashe@kivikakk.ee 2024-11-09 11:54:38 27316b9a
│ │  RmlParser: implementing in Token.
│ ○  nlrsutxv ashe@kivikakk.ee 2024-11-09 11:54:38 3e7613e7
│ │  RmlParser: strip_tokens.
│ ○  pypuqnwp ashe@kivikakk.ee 2024-11-08 16:08:40 1beaebab
│ │  RmlParser: blocks everywhere, include offset in bc.
│ ○  uvmwxovu ashe@kivikakk.ee 2024-11-08 15:29:14 b0c4bbba
│ │  RmlParser: test roundtrip.

Change nlrsutxv (commit 3e7613e7) introduces the change we’d like the test to inform, so we want to slot the test in right before then.

$ jj rebase -r @ -B nl
Rebased 1 commits onto destination
Rebased 5 descendant commits
Working copy now at: rltuvkoz 99a0a2a0 content_spec: assert body_excerpt strips tokens.
Parent commit      : pypuqnwp 1beaebab RmlParser: blocks everywhere, include offset in bc.
Added 7 files, modified 39 files, removed 8 files

jj rebase can rebase a “branch” (-b), a revision and its descendants (-s), or just a single revision (-r). -r @ means the revision currently edited in the working copy. -B means “insert before”.

The log now looks like this:

○  xnsrwqok ashe@kivikakk.ee 2024-11-09 12:02:07 cef52288
│  (no description set)
○  rvumupsn ashe@kivikakk.ee 2024-11-09 12:02:07 rml-parser* 735abf79
│  remove local artefacts.
○  ppmltptr ashe@kivikakk.ee 2024-11-09 12:02:07 f34335bf
│  Token: migrate uses of #token to #original_token.
○  pytwvkmx ashe@kivikakk.ee 2024-11-09 12:02:07 0ff90a43
│  RmlParser: implementing in Token.
○  nlrsutxv ashe@kivikakk.ee 2024-11-09 12:02:07 78390b57
│  RmlParser: strip_tokens.
@  rltuvkoz ashe@kivikakk.ee 2024-11-09 12:02:07 99a0a2a0
│  content_spec: assert body_excerpt strips tokens.
○  pypuqnwp ashe@kivikakk.ee 2024-11-08 16:08:40 git_head() 1beaebab
│  RmlParser: blocks everywhere, include offset in bc.
○  uvmwxovu ashe@kivikakk.ee 2024-11-08 15:29:14 b0c4bbba
│  RmlParser: test roundtrip.

We can now return to what we were doing, WIP ready for us to resume as we ever were:

$ jj edit xn
Working copy now at: xnsrwqok cef52288 (no description set)
Parent commit      : rvumupsn 735abf79 rml-parser* | remove local artefacts.
Added 1 files, modified 9 files, removed 6 files

Note that git commit IDs have changed, as you’d expect, but the jj change IDs haven’t. This stability of identity is very handy — xn was what I was working on before I started this aside, and it still is afterwards.

The other side of this is the rml-parser bookmark — jj’s equivalent to git’s branches, but used far less frequently (most often for git interop) — has followed its change, with the asterisk after noting it’s diverged from the remote one. You don’t have to chase down your branches after a rebase.

The manoeuvre: git

How does the same play out with git? Let’s look at the commit log from git’s point of view:

* 478c7377a4 (Fri, 8 Nov 2024) - (rml-parser) remove local artefacts. <Asherah Connor>
* 25b122abd8 (Fri, 8 Nov 2024) - Token: migrate uses of #token to #original_token. <Asherah Connor>
* 27316b9a2e (Fri, 8 Nov 2024) - RmlParser: implementing in Token. <Asherah Connor>
* 3e7613e704 (Fri, 8 Nov 2024) - RmlParser: strip_tokens. <Asherah Connor>
* 1beaebab8c (Fri, 8 Nov 2024) - RmlParser: blocks everywhere, include offset in bc. <Asherah Connor>
* b0c4bbbafb (Fri, 8 Nov 2024) - RmlParser: test roundtrip. <Asherah Connor>

We want to introduce our change — currently just an untracked change in the working tree, with develop checked out — before commit 3e7613e704.

We have two ways of getting it there:

Stash the change, interactively rebase rml-parser and stop after the previous commit (edit 1beaebab8c), pop stash, commit, continue rebase.
Commit the change now, interactively rebase rml-parser and add a pick line for our new commit in the right place.

I tend to commit early and often, so I’m more a fan of #2 as a rule, and we’re also already managing one item on the stash (our WIP from the tip of the branch), and while there’s no problem with putting as much as we want on it (it’s a very competent stack!), I just don’t wanna.

Alas, more choices:

Commit to develop, copy the commit ID to the pasteboard, hard reset develop back to its previous value.
Create a new branch, commit to that, delete branch when done.
Detach HEAD, calm your git’s nerves, it’s ok I promise¹, commit.

We don’t really want this commit on a branch, but git really wants us to want a branch. Let’s go with purity.

$ git checkout --detach
M       spec/models/content_spec.rb
HEAD is now at 83ad162d7c Merge pull request #1736 from backfill-pr
$ git add -p
[...]
$ git commit -m 'content_spec: assert body_excerpt strips tokens.'
[detached HEAD 6005753744] content_spec: assert body_excerpt strips tokens.
 1 file changed, 8 insertions(+)

Now it’s time for the manoeuvre.

$ git checkout rml-parser
Warning: you are leaving 1 commit behind, not connected to
any of your branches:

  6005753744 content_spec: assert body_excerpt strips tokens.

Switched to branch 'rml-parser'
$ git rebase -i 3e7613^

We’re presented with this:

pick 3e7613e704 RmlParser: strip_tokens.
pick 27316b9a2e RmlParser: implementing in Token.
pick 25b122abd8 Token: migrate uses of #token to #original_token.
pick 478c7377a4 remove local artefacts.

Very easy: we add pick 6005753744 above the first line, save and quit.

Successfully rebased and updated refs/heads/rml-parser.

Here’s our git log:

* 7b836073ca (Fri, 8 Nov 2024) - (HEAD -> rml-parser) remove local artefacts. <Asherah Connor>
* 70626ff5cb (Fri, 8 Nov 2024) - Token: migrate uses of #token to #original_token. <Asherah Connor>
* b4089df702 (Fri, 8 Nov 2024) - RmlParser: implementing in Token. <Asherah Connor>
* e385f17253 (Fri, 8 Nov 2024) - RmlParser: strip_tokens. <Asherah Connor>
* 652607c2f0 (Sat, 9 Nov 2024) - content_spec: assert body_excerpt strips tokens. <Asherah Connor>
* 1beaebab8c (Fri, 8 Nov 2024) - RmlParser: blocks everywhere, include offset in bc. <Asherah Connor>
* b0c4bbbafb (Fri, 8 Nov 2024) - RmlParser: test roundtrip. <Asherah Connor>

Don’t forget to pop the stash!

$ git stash pop
On branch rml-parser
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        new file:   app/lib/rml_parser_flux.rb

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   app/lib/rml_parser.rb
        modified:   spec/lib/rml_parser_spec.rb

Dropped refs/stash@{0} (d5c281196fa218a33a55538111fa7770284ca2cb)

Damn. I forgot to supply --index, and only new files (which were tracked at the time of stash) are added to the index; all other stashed changes are restored into the working copy, but not into the index. Oh well, it’s git: I can just go again with the stash reference from the last line.

Remarks

The bit that got me was that git really forces me to make a lot of decisions I don’t actually care about. How will I save my WIP while I’m off on this quest? Do I want to juggle stashes and my working tree, or throw commits around? How will I get a commit where I want it? Do I need to come up with a branch name? And how much of all this needs to just sit in my head or pasteboard, lest I forget what I was in the middle of?

It’s astonishing, too, that one of the most powerful tools git has to rewrite history is “provide a script which sequentially constructs the DAG you want, where you can insert breakpoints to manually do things you can’t express in the script”. With jj you just .. put the commit there. It was already in a commit because everything is. And when you go back to where you were, everything is still there, because it was a commit, too, just an unfinished one.

Footnotes

git config --global advice.detachedHead false ↩

Soft skills: jujutsu early feelings

2024-11-07T07:07:00Z

I finally got bothered to try jujutsu. It’s often hard to convey just how fast I’m accustomed to moving with git, and so while there are many valid complaints about its interface, object model design, etc., I’m really super fluent with it!

So for a long time I was happy to let jujutsu just be a thing people were talking about; simpler or better than git and so on, I didn’t ever quite hear anything that made learning it sound worthwhile yet.

Recently Steve Klabnik’s Jujutsu Tutorial passed by on Lobste.rs again, and it was enough to get Ashe interested. After a few days of them having a look, I bit the bullet and insisted we set up for both of us to use it¹.

It’s really good. We’ve been using it on every git repo we touch, to maximise our exposure and really put it through its paces, and it’s been so unexpectedly rewarding.

Fluent git can involve a lot of rewriting history and working on and with the commit DAG itself. jj elevates many of the involved operations to first-class status, and jj’s answer to git’s index — that the working tree represents the state of the currently-edited commit, and editing the working tree edits the commit — is the core of the mind-frame shift, like kak/hx’s pivot re: vi about command sentence order.

Things that combine to change how things work include:

The working tree reifies the edited commit, which obviates the index. This simplifies the design and allows for so many more fully-general operations.
- Practically speaking, jj snapshots the working tree at the start of every command. (Now you know.)
- As above, imo this is the core mind-frame shift required, and is what I see the most wailing and gnashing of teeth about. Do it.
Rebases always succeed, with conflicts stored meaningfully in history. Conflict resolution propagates automatically to descendent changes, which are not commits.
- It’s a little hard to describe exactly how neat this actually can be and is without your head already being in the jj model. If it sounds a bit magic, compared to the status quo, it honestly is.
git’s promise of at-least-never-losing-any-data is elevated to the level of the repository itself in the operation log.
- Sometimes knee-deep in a complicated interactive rebase, I’ll realise I’ve made a few too many wrong moves and it’s easier just to abort and start over. The data’s always safe, but on occasion you’ll need to visit the reflog to retrieve it, and manoeuvring the repo back into a particular gnarly conflicted merge state can be frustrating. jj characterises every state-modifying operation in the “operation log”, so you can instead play back and forth whatever you just did to your repo, and not just what’s in it.
- “Snapshot the working tree” is an example of an operation recorded in the oplog; it’s not magic.
If cheaper branching was a selling point for git², then branchless is the epitome of that selling point.
- Every change can be a branch. Typical git usage relies on branches to know which commits in the database should be considered “reachable” and therefore not irrelevant and to be garbage collected. jj puts the heuristic in a different place: instead of guessing which commits are relevant, record when they become irrelevant. Moving off an empty commit? Abandon it. Edited a commit, or rebased some? Abandon the old version/s. When you make a few changes on a few new commits, they all simply stay put, and you can incorporate/ rebase/abandon them later as you see fit. That’s how you accidentally an entire “stash” concept, and it’s easier to use than git stash.
- Note the implication: you might accidentally some other (perhaps new!) concepts too, just from your regular jj use evolving some patterns.

Made this post because I noticed my log currently looks like this:

@  nvxttvku charlotte@lottia.net 2024-11-05 13:19:23 8dbb3ac9
│  (empty) (no description set)
○  oypwlzvr charlotte@lottia.net 2024-11-05 13:18:23 git_head() 9d6049b2
│  Imtui: object system, sigh
│ ○  rqkpqpsk charlotte@lottia.net 2024-11-05 13:19:07 08d3afb3
├─╯  play around with https://un5t0br5cfrx6zm5.julianrbryant.com/devlog/2024/#2024-11-04
◆  ymxnqszq charlotte@lottia.net 2024-10-29 17:08:48 main fa7f5da9
│  Imtui: experiment with "generations" to GC/disregard killed components
~  (elided revisions)
│ ○  rsxxxkzu charlotte@lottia.net 2024-10-28 11:19:02 2819f0d1
├─╯  Imtui: editors cont'd
◆  wvwyouqo charlotte@lottia.net 2024-10-28 11:13:26 f4a61c28
│  Imtui: editors init

Multiple branches, without names! It’s happening.

Footnotes

tl;dr: this, where A and C run jj with JJ_USER and JJ_EMAIL set accordingly. ↩
At the time git started to really gain popularity, Subversion was popular, which itself became popular in part because it made branches so cheap. Subversion implemented branches as simply copies of a directory — typically the root would have a trunk directory, equivalent to today’s main, and to make a branch, you just recursively copy that to a subdirectory of the branches root directory. Subversion’s key improvement is that the tree isn’t actually copied in the repository; it can reuse the existing one, with deltas applied separately. Keep in mind the much smaller harddrives and lower connection speeds of the time.
This led to very busy trees, however, and commits to different branches would still interfere with each other and the mainline since they’re in actual fact all sequential commits to one big tree, not to mention the much poorer tools for dealing with branches-as-embodied-in-the-tree. When the contents of a repo are numerous or large enough, this quickly becomes an end-user cost as well: the branches may be links on the server, but on your local filesystem they’re full copies.
The costs are all doubled, too; Subversion’s primary local knowledge of the repository (remember that it doesn’t store history/isn’t a “clone”!) is kept in the form of a copy of every currently checked-out file, used as the basis for a quick status/diff to know what you have changed. So 10MB of files stored in trunk and one branch will use 40MB of space on your disk.
Branching in Subversion requires online access to the repository, permissions to create the target, local diskspace to accommodate two copies of the source, and a repository-unique name.
git’s by comparison needs none of the first three, and the last only locally. ↩

Non-intrusive vtable

2024-10-29T07:07:00Z

const Control = union(enum) {
    button: *Controls.Button,
    menubar: *Controls.Menubar,
    menu: *Controls.Menu,
    menu_item: *Controls.MenuItem,
    editor: *Controls.Editor,

    fn generation(self: Control) usize {
        return switch (self) {
            inline else => |c| c.generation,
        };
    }

    fn setGeneration(self: Control, n: usize) void {
        switch (self) {
            inline else => |c| c.generation = n,
        }
    }

    fn deinit(self: Control) void {
        switch (self) {
            inline else => |c| c.deinit(),
        }
    }
};

I’m calling this thing a “non-intrusive vtable”. A brief description for the less familiar with Zig.

In short, we have a tagged union, Control, that can store a pointer to one of the Controls types listed. Because it’s a tagged union, the variable itself stores both the pointer as well as a “tag” value indicating which of the variants is being stored, so there’s no type confusion.

(One of these is unfortunately still 16 bytes on 64-bit systems: the tag, which could theoretically be stored in 3 bits, nonetheless has a full 64 bits allocated to it, because the payload that follows is a pointer, which must be aligned. ¯\_(ツ)_/¯)

In regular Zig code you can switch on the value to get at the payload:

var b: *Controls.Button = undefined;  // pretend we have one
var c: Control = .{ .button = b };

switch (c) {
    .button => |p| {
        // this branch will run, with p == b.
    },
    .menubar => |mb| {
        // won't run in this case, but you get the idea.
    },
    else => {
        // mandatory default if all cases aren't handled explicitly!
    },
}

Now, these Controls types all carry a generation: usize member. One way of getting the generation of an arbitrary Control would be this:

fn getControlGeneration(c: Control) usize {
    return switch (c) {
        .button => |b| b.generation,
        .menubar => |mb| mb.generation,
        .menu => |m| m.generation,
        .menu_item => |mi| mi.generation,
        .editor => |e| e.generation,
    };
}

This works fine, and is probably optimal. But what’s even more optimal is letting comptime do the codegen for you. Returning to the definition above, we find:

    fn generation(self: Control) usize {
        return switch (self) {
            inline else => |c| c.generation,
        };
    }

inline else is inline in the same way inline for and inline while are. For every unhandled case, a prong is unrolled. This means the body must compile for each possible capture (i.e. each payload type). (You can of course do comptime calls here, to do different things with different kinds of payloads, though please consider your complexity budget!)

In this way, we create dispatch functions that encode the knowledge of (biggest air quotes in the world) “all their subtypes’ implementations”. Ha ha ha.

Aside on tags

As a kind-of-side, another neat thing Zig affords when working with tagged unions is call-site ergonomics when passing them to and from functions.

We have Controls stored in a hashmap with a string lookup. Here’s a first version of a “get control by ID” function:

fn controlById(id: []const u8) ?Control {
    return controls.get(id);
}

Too easy. But now using it looks like this:

fn render() {
    if (controlById("menubar")) |c| {
        const mb = c.menubar; // <-- safety-checked variant assertion
        // mb has type *Controls.Menubar.
    }
    // ...
}

And indeed, most actual uses will probably need an intermediate, since most call-sites will actually know the type of what they’re asking for. Why not instead roll that into the getter?

fn controlById(
    comptime tag: std.meta.Tag(Control),
    id: []const u8,
) ?std.meta.TagPayload(Control, tag) {
    const c = self.controls.get(id) orelse return null;
    return @field(c, @tagName(tag));
}

std.meta.Tag(Control) gets the type of the “tag” type of the given type. What a mouthful. In other words, Control is a tagged union, and the tag is the implicitly-defined enum that represents which variant is chosen. For Control, that enum takes the values .button, .menubar, .menu, etc.

We take the expected tag of Control at comptime, and declare our return type to be (the optional of) the payload in Control that corresponds to tag, using std.meta.TagPayload(Control, tag). So a call like controlById(.button, "xyz") has the return type ?*Controls.Button.

(Zig will unroll one of these functions per distinct tag, and so type-errors may occur if something doesn’t match up for one particular case! Conversely, you can write code that wouldn’t check for all variants if you don’t plan on using it for those, but it’s a friendly act to yourself and others to write explicit checks with explicit messages. :)

We fetch the Control as before, but now we use the @field builtin to perform the equivalent of c.blah, where blah is specified by @tagName(tag).

The @tagName builtin returns a comptime string of the tag’s name, so for .button it gives "button". This is what @field wants, and so the effect is a safety-checked variant assertion, like we were doing in our “user” code before, only now it’s done as part of the getter itself.

If you wanted to, you could even make the function non-asserting, instead returning null if the type was a mismatch:

fn controlById(
    comptime tag: std.meta.Tag(Control),
    id: []const u8,
) ?std.meta.TagPayload(Control, tag) {
    const c = self.controls.get(id) orelse return null;
    return switch (c) {
        tag |payload| => payload,
        else => null,
    };
}

You get the idea! You can do loads with this stuff.

VyxOS

2024-07-19T11:27:00Z

Just a little note to say that I’ve published VyxOS, my Nix configuration.

It’s just over a year old at this point, and at one point supported a large mesh of eccentric servers with its own dynamic mesh and DNS and all sorts of nonsense, but the config and infra has been hugely condensed to make it friendly to my brain, and the history scrubbed to make it friendly to publishing.

Hope it’s interesting!

Time travel, raw

2024-07-06T16:21:00Z

Raw log from my notes re: Time travel follows.

Sae

RV32I with some RV32C/refactoring WIP from long ago. The WIP probably feels way too magic for me now, but we should take a look at it. Now uses Niar.

TODOs

Decombing
RV32C and associated refactor
- Then add RV32E, RV64I?
The entire test infra could be so much more robust.
M extension
A extension
“Zicsr” extension
BMC (WHAT DID THIS MEAN)
ili9341spi interface

Decombing

First priority is decombing the design to try to get the build time down. It’s currently redonkulous:

[2024-07-03 13:05:24,917] niar: INFO: building sae for icebreaker
[2024-07-03 13:05:24,917] niar: DEBUG: starting elaboration
[2024-07-03 13:05:25,148] niar: DEBUG: elaboration finished in 0:00:00.230441
[2024-07-03 13:05:25,148] niar: DEBUG: 'sae.il': 425,987 bytes
[2024-07-03 13:05:25,148] niar: DEBUG: starting synthesis/pnr
[2024-07-03 13:05:25,148] niar: INFO: [run]   execute_build
[2024-07-03 13:08:12,179] niar: DEBUG: synthesis/pnr finished in 0:02:47.031564
[2024-07-03 13:08:12,207] niar: INFO: 
[2024-07-03 13:08:12,207] niar: INFO: === sae ===
[2024-07-03 13:08:12,207] niar: INFO: 
[2024-07-03 13:08:12,207] niar: INFO:    Number of wires:               2859
[2024-07-03 13:08:12,207] niar: INFO:    Number of wire bits:           9313
[2024-07-03 13:08:12,207] niar: INFO:    Number of public wires:        2859
[2024-07-03 13:08:12,208] niar: INFO:    Number of public wire bits:    9313
[2024-07-03 13:08:12,208] niar: INFO:    Number of ports:                  4
[2024-07-03 13:08:12,208] niar: INFO:    Number of port bits:              4
[2024-07-03 13:08:12,208] niar: INFO:    Number of memories:               0
[2024-07-03 13:08:12,208] niar: INFO:    Number of memory bits:            0
[2024-07-03 13:08:12,208] niar: INFO:    Number of processes:              0
[2024-07-03 13:08:12,208] niar: INFO:    Number of cells:               5732
[2024-07-03 13:08:12,208] niar: INFO:      $scopeinfo                     19
[2024-07-03 13:08:12,208] niar: INFO:      SB_CARRY                      452
[2024-07-03 13:08:12,208] niar: INFO:      SB_DFF                         79
[2024-07-03 13:08:12,208] niar: INFO:      SB_DFFE                        35
[2024-07-03 13:08:12,208] niar: INFO:      SB_DFFESR                    1380
[2024-07-03 13:08:12,208] niar: INFO:      SB_DFFSR                        8
[2024-07-03 13:08:12,208] niar: INFO:      SB_GB_IO                        1
[2024-07-03 13:08:12,208] niar: INFO:      SB_IO                           3
[2024-07-03 13:08:12,208] niar: INFO:      SB_LUT4                      3737
[2024-07-03 13:08:12,208] niar: INFO:      SB_RAM40_4K                    18
[2024-07-03 13:08:12,208] niar: INFO: 
[2024-07-03 13:08:12,208] niar: INFO: Device utilisation:
[2024-07-03 13:08:12,208] niar: INFO:            ICESTORM_LC:  5033/ 5280    95%
[2024-07-03 13:08:12,208] niar: INFO:           ICESTORM_RAM:    18/   30    60%
[2024-07-03 13:08:12,208] niar: INFO:                  SB_IO:     4/   96     4%
[2024-07-03 13:08:12,208] niar: INFO:                  SB_GB:     5/    8    62%
[2024-07-03 13:08:12,208] niar: INFO:           ICESTORM_PLL:     0/    1     0%
[2024-07-03 13:08:12,208] niar: INFO:            SB_WARMBOOT:     0/    1     0%
[2024-07-03 13:08:12,208] niar: INFO:           ICESTORM_DSP:     0/    8     0%
[2024-07-03 13:08:12,208] niar: INFO:         ICESTORM_HFOSC:     0/    1     0%
[2024-07-03 13:08:12,208] niar: INFO:         ICESTORM_LFOSC:     0/    1     0%
[2024-07-03 13:08:12,208] niar: INFO:                 SB_I2C:     0/    2     0%
[2024-07-03 13:08:12,208] niar: INFO:                 SB_SPI:     0/    2     0%
[2024-07-03 13:08:12,208] niar: INFO:                 IO_I3C:     0/    2     0%
[2024-07-03 13:08:12,208] niar: INFO:            SB_LEDDA_IP:     0/    1     0%
[2024-07-03 13:08:12,208] niar: INFO:            SB_RGBA_DRV:     0/    1     0%
[2024-07-03 13:08:12,208] niar: INFO:         ICESTORM_SPRAM:     0/    4     0%
[2024-07-03 13:08:12,208] niar: INFO:

After moving the fault check out of fetch.resolve: 1:47, 4825 LCs.
After using .all(): 1:44, 4802 LCs.
After fixing our IL digest behaviour: priceless.

After splitting out just OP_IMM: 404k IL, 2:36, 5038 LCs. O_o
I guess I need to split out the decode a little more? Or maybe it’s just a matter of decomposing more.

After replacing multiple m.d.sync += self.write_xreg(v_i.rd, ...) with one of those and a comb wire out for the value: 404k IL, 1:35, 4851 LCs.

We’ll split it out as much as possible at first, and then slowly reintegrate. We already do the register save in fetch.init, and now with some care after splitting out OP_IMM it’s a bit better again.

Need to remember that the toolchain does much less deduplication than we assume. Keep going on that, esp with insn decode.

Using ~insn[:16].bool() instead of == 0: 404k IL, 1:52, 4800 LCs.
Using wb_reg.any() instead of != 0: no change.

After splitting out LOAD: 404k IL, 1:59, 4945 LCs. Uhm.
After factoring the xreg fetch into common: 402k IL, 1:46, 4972 LCs. Hmmmmmm.

After adding the read register: ran out of BELs. Welp. (6515 cells.)
After changing the read register comb->sync: 6394 cells. Improved slightly.
After splitting out OP: 371k IL, 7166 cells. …
After refactoring OP with out: 7120 cells.
After splitting out STORE: 368k IL, 7134 cells.
After splitting out BRANCH: 343k IL, 6386 cells.
After splitting out JALR: 339k IL, 5599 cells and PNR is working again.
After changing jump(m, pc)’s context manager return to ~_.bool(): 339k IL, 5602 cells. Uh, ok. Reverting that for now just ‘cause maybe there’s a cross-over point (size of bv).

Using any() instead of bool() causes cell reduction? At 5587. 339k IL, 1:04, 4774 LCs.
OK, switching sync->comb on read regs bumps back up to 5664 (+77), and increases PNR time significantly (maybe because we’re close to cell count?). 338k IL, 2:41, 4984 LCs (94%).

Next step is to do the instruction decode in one place and then pass info to following stages.

Added imm and funct3 to LOAD: 342k IL, 1:06, 4905 LCs (92%).
Did the same to OP_IMM: 342k IL, 0:58, 4886 LCs (92%).
Removed v_sxi wire and just used imm[:12].as_signed() in place: 342k IL, 1:01, 4907 LCs (92%).
Did same deal to OP: 343k IL, 1:01, 4806 LCs.
Did same deal to STORE: 344k IL, 1:16, 4816 LCs. !!!
I forgot to only use the bottom 12 bits of imm. Fixed: 344k IL, 1:09, 4910 LCs. What?
Try doing the sign-extension in resolve: 344k IL, 1:13, 4845 LCs.
Do the thing for BRANCH: 337k IL, 1:20, 4859 LCs.
JALR too, that’s everything: 337k IL, 1:13, 4812 LCs.
Drop v_sxi and do the sign extension in resolve: 336k IL, 1:15, 4825 LCs.
Same for LOAD: 336k IL, 1:17, 4774 LCs (90%). Huh.

op.op_imm and op.op can be refactored.
Hackily done: 333k IL, 0:59, 4595 LCs. OK yeah, that helps!
Done so that it actually works (still hack): 333k IL, 1:05, 4633 LCs (87%).

Put register file in memory now that it’s all separated out.
(How much is it using, really? Half XCOUNT: 288k IL, 0:19, 3340 LCs (63%). OK, quite a bit.)

Pico fits in 750–1000. SERV fits in 198??????

Dumped it in a register file. Gave it two read ports so no existing code has to change, I think it’s just duplicated the memories but they’re so small it doesn’t matter. 265k IL, 0:13, 2367 LCs (44%).

Cleaned up our reg read and write logic: 264k IL, 0:14, 2300 LCs (43%).

TODOs remaining:

Read all the accepted Amaranth RFCs.

OK cool.

Do a once-over and generally clean up the Hart.

2291 LCs.
2162 LCs after combing the MMU interface.
2269 when I do it to the MMU write port. No point since we have to register a lot anyway. Back to 2162. Similarly it grows when I use comb to set req_width — maybe because everything else in the FSM (back at this point) is sync, so they all switch together. Hrm.

Let’s try changing the write_xreg to comb. Basis: 8e3c38ca, 2162 LCs. Hm, nah — this can’t work. We assemble the components over multiple cycles fairly often (use xwr_reg to store v_X.rd etc.). What about read_xreg? The result is we must read the result when expected, since we’re no longer registering the address. I anticipate a growth in LCs (but faster reads).

2266 LCs, but haven’t removed the extra states yet so tests all fail.
2303 LCs, necessitated an ALU refactor. So I don’t think there’s a benefit to this other than speed? Let’s see how many cycles CXXRTL gains.
57508 after change, 57505 before. Well! The ALU split is something though, if I keep that it’s gonna be uglier anyway. How can I fix up?

Maybe we can tell the ALU where to read its inputs.

Adding a top-level “delay” that gates the whole FSM adds 100LCs. It’d be nice to have one wait state. Ah well.

2230 LCs after centralising the ALU. Splitting it into two stages gets us to … 2246? Oh. OK. Really didn’t expect that. Guess I won’t do that.

If we don’t have reg read always on, we can actually shuffle bits like imm into xrd2_val.

Up to 2303 on adding xrd[12]_en. … and hold the phone, xrd2_val is comb-driven. Nvm.
It barely helps us anyway since we need to post-process for the ALU. Cancel that.

2202 after dropping the m.If(funct7[5]) out of the Else() block — we know they’re mutually exclusive, whereas it doesn’t.

On the contrary, 2189 after changing the m.If(funct3 == …ADDSUB): … / m.If(funct3 == ….SR) to if/elif: easier mux? It resembles a switch (which are probably optimised …).

2202 after collapsing l.wait states. Cleaner. 2194 after fixing the default bug — that’s 41%.

I would love to get a better idea where all these cells are being spent, but it’s pretty hard to say after optimisations.

Consider the same for the MMU.
- At minimum, use amaranth.lib.stream for its interface.
Stop embedding the “address bus” in the MMU along with the UART hook.
Clean up UART.
Move onto the big task.

Aside: ABTest

I feel like I want to make a little sandbox or something that makes evaluating the RTLIL diff between Amaranth expressions easier (optionally running it through opt or all the way through synthesis). Ideally it could even run in-situ, i.e.

with ABTest.A():
    m.d.comb += blah.eq(x[:16] == 0)
with ABTest.B():
    m.d.comb += blah.eq(~x[:16].bool())

Or even:

m.d.comb += blah.eq(ABTest(x[:16] == 0, ~x[:16].bool()))

All such sites would be toggled individually (with others defaulting to “A”, no cartesian product) and then outputs presented for comparison.

Sae’s a bit slow to try this with right now.

RV32C

This will take some re-understanding. We know the shape of the ISA(s) better now so we might be able to design something less Heppin Magic.

The cherry-picking went fairly straightforwardly, lots of conflicts but all easily resolved. Glad we did it in this order!

There’s so much magic in isa.py that I’m resolved to redesign this in a much more straightforward way.

Design

Users of an ISA defined with this tool:

assembler/disassembler
- Opcodes and registers accessible via reflection, including support for defining shorthands (J, LI, etc.).
- Need to be able to go the other way, too.
gateware
- Clear and easy access to layouts and op constants.
- Exposes metadata like ILEN/XLEN/XCOUNT for gateware to use.
subclassing ISAs
- Can be added to, removed from (e.g. RV32E reducing XCOUNT).

Goals:

Much less magic.
- Avoid metaclasses, avoid __call__.
- Inspecting signatures is OK
Just enough flexibility to express RV; other ISAs are currently a non-goal.
- I think the current design encapsulates most of what we’ll need here.

Notes on the existing design:

Our current design doesn’t include an intermediate representation: IThunk.__call__ winds up calling shape.const().as_value().value, building up args to pass to const(); there’s no real point of “calling it done” except for the first time it’s called.
Many layouts define immediates in groups of imm0_4, imm5, imm6_11 kinds of things. Sometimes they also omit e.g. imm0 (implied 0).
An ISA has a notion of a Register, which is a class defined in the return value of ISA.RegisterSpecifier() (!).
- This uses locals().update(members) to define the members of an IntEnum, where registers is built up from a list of (name, alias0, …, aliasn) tuples and a target size.
- I think we’ll still need something like this; it’s actually one of the least magic parts of this.
All ISA members can define _needs_named and _needs_finalised attributes, processed in ISAMeta.__new__.
- _needs_named causes the assignment of __name__ and __fullname__ attributes, according to the name being assigned to.
- _needs_finalised calls finalise on the object with a reference to the ISA.
- This lets members finish initialising themselves with an awareness of everything else defined in the ISA, including things defined (lexically) after them.
- ILayout is an empty baseclass with an ILayoutMeta metaclass.
- ILayoutMeta.__new__ takes an optional argument len and assigns it to cls.len (where cls is the newly-created class).
- If layout is specified, it marks the class as needing finalisation and checks that cls.len is in fact defined (either now, or in a superclass). Otherwise, it’s considered a layout base class.
- ILayoutMeta.finalise:
  - assembles the defining context dictionary by iterating the ISA’s MRO backwards for their dir()s, discounting names starting with underscore (_);
    - In other words, items in the ISA class and superclasses define the context for type-shape lookups.
  - assembles the full type-shape dictionary by iterating the ILayout instance’s MRO backwards for annotations, starting from after ISA.ILayout itself;
    - In other words, annotations in class and its ISA.ILayout superclasses define the set of type-shapes available to layout items.
    - The context dictionary is used as locals() here.
  - iterates over the layout tuple given by the subclass, constructing cls._fields by matching names to ShapeCastables:
    - Members can be strings, in which case they refer to an annotation with a matching name.
      - If the exact match lookup is unsuccessful, the class’s resolve() function is called with some context (the remaining items in the layout, length of instruction remaining needing to be allocated to a field), which must succeed.
    - Members can be (name, shapecastable).
  - initialises cls.shape = StructLayout(cls._fields).
  - initialises cls.values and cls.defaults by calling resolve_values on the class’s existing (set by subclass definition) values and defaults members, if any.
    - These may not overlap.
    - ints are ints, strings are treated as keys for the ShapeCastable for the corresponding field.
      - If item lookup fails, the ShapeCastable’s __call__ is tried.
- ILayoutMeta.resolve just raises an error. Is this really exposed on subclass instances? Surprising.
- ILayoutMeta.xfrm constructs the class and calls xfrm on it.
  - If I is an ILayout subclass, this just means I.xfrm(…) is the same as I().xfrm(), i.e. get an unrefined thunk and then transform it.
    - Digression: for whatever reason we really like being able to use classes in these positions. It “must” be a class because it’s the result of defining something with class Blah:, which itself is needed because we often want to supply code, nested classes, etc. But why the insistence on calling the class itself? We don’t ever have class instances, and doesn’t that seem a bit strange?
    - Thinking forward, the class instances should be the intermediate representation, not a separate thunk class. You call I() or I(a=b), you get a <myisa.MyISA.I object>, with the args hitting I.__init__ like a regular human being.
    - This prevents our delightful (…) hack with I(s). We can actually just call it I.shape(s), which already exists because that’s what it does lol!!
    - I have some lingering concerns here around repeated work that currently happens in finalise etc. but let’s deoptimise now, and reoptimise after the design is sane.
- ILayoutMeta.__call__ allows zero or one positional arguments, plus kwargs.
  - In the above example, this is I(…).
  - Zero positionals asserts a layout is defined, and returns a new IThunk(cls, kwargs).
    - In other words, "I(a=b)". This denotes a partially refined instruction based on I.
    - Note that even I() is valid syntax, to get the same kind of thunk but not refining any part of it.
  - One positional asserts a Signal argument is given and wraps it in the subclass’s shape (cls.shape(s)), so you can call I(s) to decode s.
- The IThunk is as close as we get to an “intermediate representation” here.
  - Sets _needs_named, as it’s probably going to be assigned in an expression like ADDI = I(funct3=I.IFunct.ADDI).
  - Stores the class it was constructed from and the kwargs we got.
  - xfrms initialised to empty.
  - asm_args is defined from list(self.layout): it’s the list of arguments an assembly call need to provide. If your layout is ("opcode", "rd", "imm"), we need an opcode, dest register and immediate.
    - opcode is defined as type Opcode and rd as Reg in the defining context, and imm is handled in IL.resolve when it’s in the final position.
    - The opcode is refined by being specified in kwargs, leaving just rd and imm for the “asm args”. So how does that happen?
  - We iterate over all values and defaults in the IL class, and names in kwargs, removing from asm_args any specified there.
  - Next we iterate names in kwargs, asserting all specified are a part of the layout, and none are part of the IL class’s values (the distinction between values and defaults being whether they can be overridden in a thunk ctor or not).
- It has clone() and partial(**kwargs); the former returns a new IThunk with copies of all settings (for declaration, immutable definition), the latter clones and updates clone.kwargs with given kwargs, removing those from clone.asm_args (further refinement of an IThunk).
- It also has xfrm(xfn, **kwarg_defaults), which appends a new transform to clone.xfrms, with some optional default kwargs.
  - Transforms are a function which are handed a set of kwargs, and return a dict to update kwargs given to the next one (or to the ilcls.shape.const(…) call at the end).
  - The kwargs start out as the thunk’s own kwargs mixed with any given to the IThunk.__call__, latter superseding the former.
  - The transform function’s signature is analysed: if you take a parameter x, the kwarg x is filled in (mandatory). If you specify x=default, then kwarg_defaults and finally default are used as fallbacks.
    - I wonder why kwarg_defaults is only allowed when no default is given. I guess they’re either really mandatory to specify, or possibly optional.
    - An example here is shamt_xfrm(shamt, *, imm11_5=0). SRAI overrides this with SRAI = I(funct3=I.IFunct.SRI).xfrm(I.shamt_xfrm, imm11_5=0b0100000); the others don’t override it at all.
    - In other words, kwarg_defaults is more like “default overrides”. In either case I don’t imagine a user is actually setting one in a thunk, so maybe they should be treated that way.
  - What’s unspecified here is a way for transforms to also transform asm_args, and that’s where I got up to with# clone.asm_args. ## RESUME XXX GOOD LUCK.
- When an IThunk is called, we resolve the args_for the given kwargs.
  - We call the transform pipe with self.kwargs | args, i.e. those given while constructing the thunk mixed with those given while calling it.
  - The result of the pipe is asserted to match the layout and not override anything it’s not allowed to override.
  - The ilcls.values, ilcls.defaults (both already ‘resolved’) and result of resolving the pipe’s output are all combined and become the args passed to shape.const.
- Note that transforms are called in the order given, so we must transform asm_args back-to-front, as inputs used by earlier transforms may be provided by later ones.
  - Actually this is just backwards unless we do yet-more-thunking/accumulating. Let’s reverse the order of how it should be called, so we can apply asm_args changes as xfrm() is called repeatedly. Actually call the transforms in reverse order.

Time travel

2024-07-06T14:07:00Z

The typical hypothetical “who are you coding for” example meant to shock you into writing better code is “yourself in six months”, but it turns out four is completely adequate to get lost.

Start

In February I started writing my first RV32 core, Sae. By the end of the month, I had enough of one going to run some sample code compiled with GCC for RV32I, interacting with UART via MMIO, which was such a nice feeling. It was written in a very dumb manner — meaning it used 95% of the UP5K and took 3 minutes to synthesise — but on the other hand it ran pretty much at one cycle per cycle! (I hadn’t pipelined instruction memory fetches, so it didn’t, but that was the only thing standing in the way of that. It was written very, uh, plainly.)

Next I wanted to tackle RV32C, and one thing I wasn’t happy with was how I was actually defining the ISA. It was very manual, and it felt like there was a lot that could be extracted out in less ad-hoc ways, so that I didn’t then have to repeat myself even more when adding RV32C, or e.g. making it possible to define a thing called “RV32E” (which is just RV32I with 16 X registers instead of 32) as a refinement of RV32I instead of repeating a whole bunch of code, and then making that selectable as the basis for a core/hart. This also provided an opportunity to expose more metadata for the built-in assembler and disassembler used by the test suite.

I started to extract a bit of a “construction kit” for ISAs, and as was my Python style at the time, there was a lot of magic involved. One of the commits describes redoing something “with just slightly less magic”, but iunno, a lot of magic follows in the commits after that one.

And then, on March 5, a small “wip” commit is the last of the branch, with some code that’s kind of half-there, half-nonsense, and the following intentionally syntactically incomplete line:

             clone.xfrms.append(pipe)
+            clone.asm_args. ## RESUME XXX GOOD LUCK

Change

At that stage I was about a month off moving to Estonia, and so all work stopped. By the time I started to do HDL again, two months later, I’d decided to stop using Amaranth entirely and learned to do digital design in Chisel, which was a big undertaking. A lot of the machinery around the design needed to be implemented — connecting to resources on IO pins, the concept of building for different platforms, etc. — a lot of the toolchain needed to be replaced/rebuilt, but the actual manner of design changed a lot too. I’d taught myself HDL with Amaranth, and I sort of retaught myself it with Chisel, grasping the fundamentals more clearly.

(I also wrote a framework for Chisel in this time, Chryse, which is where all my non-project-specific bits went, including the replacements for what Amaranth gave outside the HDL itself. I had a project framework for Amaranth, Rain, so Chryse was like Amaranth build’n’platform plus Rain.)

Three-ish weeks ago, I started using Amaranth again, and it was time to see where my old work was at. First, I ported everything new from Chryse back to Rain, freeing it from its weird Nix flake and so giving it a new name, Niar. Then I ported my last Chisel project back to Amaranth, ili9341spi, using it as the basis for testing Niar. (It’s just a TFT LCD controller, rendering Conway’s Game of Life.)

Once I’d gotten things to my satisfaction, it was time for my next project. I’d remembered Sae as my last big project before moving/switching off Amaranth, but my memory didn’t extend much further than that — except maybe the atrocious build times.

My next project wants a little CPU core of some kind in it, and I decided to be responsible and actually bring Sae back to life with my new sensibilities, instead of just starting fresh and having all that work sit there.

Necromancy

There was the main branch, last modified February 28, and then 27 commits of WIP on the rv32c branch ending in the RESUME XXX GOOD LUCK comment. I took one look at the code in that branch and I realised the comment was extremely apposite. I didn’t have a clue what any of the existing magic was doing, and I certainly didn’t understand what the final half-commit was driving towards either. The tests were extremely broken, and so there was nowhere really to get a foothold on it.

So it was back to February 28. First I ported it to Niar, which was uneventful, and then added a test using CXXRTL to run the same thing we build for the actual FPGA, which is this delightful little program. Synthesis still takes 3 minutes, and it turns out all but 10 seconds of this is in place-and-route. CXXRTL elaboration and compile is much faster, and running real RV code gives me more certainty than the (extensive but still artificial) unit tests that I haven’t broken anything.

At this stage I could’ve considered getting back onto the WIP, but the core gateware design issue was pretty pressing: no matter how good an experience I have of specifying an ISA, if I only have 260ish gates free, I’m not doing much with it, and the build time sucks even if I can mostly test without it. (I don’t want to be discouraged from putting it on the FPGA!)

[2024-07-03 13:05:25,148] niar: DEBUG: starting synthesis/pnr
[2024-07-03 13:05:25,148] niar: INFO: [run]   execute_build
[2024-07-03 13:08:12,179] niar: DEBUG: synthesis/pnr finished in 0:02:47.031564
...
[2024-07-03 13:08:12,208] niar: INFO: Device utilisation:
[2024-07-03 13:08:12,208] niar: INFO:            ICESTORM_LC:  5033/ 5280    95%

So instead I spent time fixing this, which meant separating the core out into stages, defining an ALU instead of repeating the ops everywhere, putting the registers into BRAM (that was 40% of the UP5K’s LCs right there), and generally making the thing slower at runtime and tightening up some definitions in exchange for 80x faster PNR and using a third of the logic resources. It’s at least somewhat amenable to pipelining, which is something I’m going to enjoy implementing later.

I proceeded methodically and generally did before-and-after builds of every change, comparing the bytecount of the generated RTLIL for a rough idea of “how much raw IL does the Amaranth input given produce”, synthesis plus PNR time, and comparing ICESTORM_LC counts of the final design (or just cells reported by Yosys where PNR was failing because the design got too big).

This is extremely fraught and open to misinterpretation, because Yosys’ output is optimised, and with many passes involved, there is no direct link between a given change and the effect on output size. PNR is similar, especially when up against resource limits, and so the whole thing requires investing not too much importance in any one result. You’re just kinda vibing it out, making hypotheses about what might be cheaper post-opt, and then, usually, trying to reason to yourself why it wasn’t actually cheaper. There’s a lot of holding of judgment because maybe you need to change all the things of class X before you can observe the true difference between two methods.

A play-by-play of this whole process is included in the raw log with the subheading “Decombing”. At the entry to this stage, synthesis took 2:47 and PNR completed with 5033/5280 (95%) of ICESTORM_LCs used. We finished at synthesis taking 0:10 and using 2194/5280 (41%).

Chrononautics

Now we were in a position to start understanding the refactor WIP.

Step one was rebasing the old branch onto the new. I’d refactored and moved things around a lot (including e.g. using Amaranth’s new async testbench stuff), so any commit that touched the existing gateware or testing infrastructure conflicted, but these were nice little opportunities to understand pieces of what I was doing in context. The branch itself had a lot of nice refactors in other areas too, so by the time I’d finished rebasing, I felt in a fairly good position, except for the fact that neither building it nor running any of the tests worked because there was a syntax error in the core ISA definition machinery.

The next step was to get it running, because I couldn’t hope to complete whatever it was I was doing in March without understanding any of its context — namely the entire new ISA kit — and understanding a system is so much easier when you can see it running.

To start with, I just commented out the broken line, and it ran, with 44/64 tests passing. Without much in the way of understanding everything around it, I was able to identify something one specific case needed to happen right at that same spot, put together an ultra-specific hack in situ, and we got to 54/64. At this point, I kind of understand what I was going for, but now I need to understand everything else to know how to make it happen.

The way to do that turned out to be just reading the new code over and over again, outlining it in natural language from the top down. The result of this process is included in the raw log under the subheading “Design”, and by the end of it, I’d recapitulated every decision made in coming up with it. The next steps I’d imagined followed naturally, and although I entered this stage with no intention to continue to use this design (on account of its magic-ness), it still made sense to me to finish my last WIP before turning my mind to simpler design, since without a fully proven concept I can’t say I’ve grasped it enough to actually design it again. This got us to 63/64 tests, with the last one being the first (prospective) RVC test.

More than proving the concept, though, I just really enjoyed finally completing that change, over 4 months later, in a completely different place and time. Continuity in the face of countless discontinuities.

Incidentally, this branch doesn’t include much in the way of actual RV32C support, since I got completely caught up with designing the new ISA kit using the existing RV32I support as a basis. I think I won’t bother with that, though, since it’ll likely entail further support from the ISA kit, and in the current design that means even more magic. Time to simplify.

Raw notes follow.

Amaranth to Chisel

2024-05-09T14:45:00Z

edit: A lot of the following doesn’t apply any more, though it’s all been very helpful in learning.

My days of using Amaranth are over. I don’t feel able — nor do I want — to depend on something I’m not allowed(!) to contribute to, so I need a way to continue on with my FPGA studies without it. I don’t really view just trying to cobble together Verilog as viable for me right now; I’m rather dependent on having a decent higher-level thing going, and I already feel all the wind sucked out of my sails from having to make any change whatsoever.

I’ve experimented with doing my own HDL using what I learned from working with and on Amaranth (and Yosys, which I’ll happily continue to depend on), but it’s way too much work. After surveying the scene, I’ve chosen Chisel. Scala is not exactly my favourite, and this means really learning it properly, but y’know what? That’s how I felt about Python too, but I still did some cursed stuff with it!

I plan to bootstrap my way out of this hole by creating a small component in Amaranth, workbench it using CXXRTL, then duplicating that component in Chisel, using the same CXXRTL workbench to test it. This way I’m staying connected to “doing useful/measurable stuff” in a way I know. I’m also furthering my own HDL experiments while I go, letting Amaranth and Chisel combine in my head.

Done so far:

Bring hdx, rainhdx, and all their dependencies — including Amaranth — up to date.
- New abc revision.
- Amaranth depends on a newer pdm-backend, which I needed to package since it’s not in nixpkgs.
- Had to unbreak rainhdx’s Nix, that last refactor was bad.
Add basic cxxsim support to rainhdx. This was mostly pulled from I²C, oh! Big stretch, which I maintain is impeccably named.
- There was also the option to pull the Zig–CXXRTL support from sh1107, but the extra toolchain weight doesn’t feel like it helps me move any faster here.
A basic UART echo, tested with Amaranth’s simulator.
A clone of the Python simulator with CXXRTL.
Learn to do a very basic Chisel module with tests and Verilog output.
Build the Chisel module with CXXRTL and integrate it into the simulator — it’ll be very wrong, but the key is the integration.
Write a little unbuffered UART pair, test them, integrate. Done.
Extend the test case to exercise the Amaranth version’s buffers on TX/RX.
~~Write a FIFO in Chisel and buffer the TX/RX.~~
Discover Queue and learn how to use Decoupled – use that in RX and TX.
Redo the base UART module using Queue.
Test it on the iCEBreaker!
Mess around with SB_RGBA_DRV. Buffer the clock input with SB_GB.
Drop all the Python; it’s no longer necessary.
Actions CI for unit tests, cxxsim, synthesis.
Introduce a “Platform” notion to build separately for iCE40 and CXXRTL.
Split off the project-independent bits.
Redo the testbench to have the test unit as a blackboxed instance, rather than it driving everything through lines from the top. ~~Get it working first with Amaranth, then Chisel.~~

And now, 12 days later, I’m done! I have a fair bit more ground to re-cover in terms of (a) actually putting together more complex designs — I’ll start with an SPI OLED (maaaaaybe with a Zig counterpart, like sh1107) and then move onto a RISC-V core again — and (b) creating my own framework to iterate on different projects quickly, but I’ve moved really fast¹ and I’m quite happy with it.

It’s been interesting to decompose “HDL and digital design, as I learned it through Amaranth” — the way I write things has become a lot more fluent, a lot less “eighty stage FSM”, and I spend a lot more time looking at Verilog, which I think is a good thing right now. I grok a lot more of what’s under the covers, especially having to reimplement some of it.

I can re-archive all my Amaranth stuff again, now that I’ve finished leaning on it.

Footnotes

With minimal tools! At the start of these 12 days I didn’t even have a desk, or an external monitor. Or a dev board! ↩

Comrak on Akkoma

2024-01-02T04:38:00Z

First up: I’ve never done more than toy with Elixir before, and never with Nix or Rust, so this “simply stuff Nix, Elixir and Rust into a magic hat” trick was guaranteed to be at least a little bit Fun™. And it was! :)

Stock Akkoma uses Earmark, which looks like a lovely library, but maybe a lil out of date and out of step with CommonMark/GFM. We deserve Comrak.

Happily enough, a Google search revealed a Nathan Faucett had already done most of the hard work of using Comrak from Elixir in ex-markdown. Thank you! This never gets old.

Ported it for Comrak and Rustler changes in the last 5 years, and then learned about the various ways to juggle Elixir and Mix releases/deps in Nix. Several hundred lines of hack-ish later and ex-markdown was now fit for purpose.

Special care was taken to ensure both nix develop- and nix build-based builds work — this one always wants to be able to quickly iterate in my checkout without waiting all day for non-incremental builds, but at the end a nix build should:

match the behaviour of a clean nix develop --command bash -c "mix deps.get && mix test";
always cleanly succeed; and,
run mix test itself as a post-install check so we don’t get blindsided by differences in the dev shell/closure-built artefact only when later using it (i.e. in Akkoma).

This required some finesse: we want to build the native Rust dependency as usual when doing nix build, which means doing the usual Cargo/Nix dance and compiling that artefact as its own derivation (and all its crate deps as their own, etc. etc.). On the other hand, in nix develop we want the usual compile-on-demand to happen. Happily, Rustler is portable enough to support this workflow! (see the MARKDOWN_NATIVE_SKIP_COMPILATION env var.)

One tricky thing is the fucken Mix dependencies. The ex-markdown derivation itself needs to introduce its own Mix deps to beamPackages.buildMix so it can actually build and test. But that’s no good when we’re building Akkoma — we want to use a release-wide resolved version of those dependencies, with all BEAM deps in the one closure and no overlap.

For now we hack it somewhat, and reproduce some of ex-markdown’s derivation in our Akkoma fork — beamPackages doesn’t have anything like overrideBeamAttrs or overrideMixAttrs at the moment. There’s a fair bit more Nix involved therein.

We started with upstream Nixpkgs’ Akkoma package definition (again, copying the original as a base due to lack of override), add our :markdown package to the mixNixDeps — we pull the source, native package and toolchain deps through the ex-markdown flake :)! —, adjust the call-sites, and then as a cherry on top, expose a NixOS module that sets config.services.akkoma.package to the package exposed. Using the new Akkoma in my personal config is as simple as referring the module.

And there you have it!

Future work

ex-markdown only used a native call for parsing the input; the rendering is done in Elixir. Let’s add the missing NIF for rendering too!
Working out a nicer way to share the ex-markdown derivation for use in downstream projects’ mixNixDeps.
Working out a nicer way to override some properties of Nixpkgs’ Akkoma derivation.
Unify version numbers and revisions.
I’ve just noticed below that Comrak’s (GFM-compliant) autolink feature breaks remote user refs by turning them into mailto’s! Oops.

Reflections

Having never really touched Elixir much, this was a reasonably intimidating circus of interdependent parts to dive right into. It was super fun and — as usual — I credit Nix with making this at all possible, and more importantly worthwhile. The fact that I don’t have to worry about accumulating platform tools (or getting them installed on the target server etc.) is only a small part of it.

I did indeed spend quite a while fucking around with Making All This Shit Work With Nix, but I’d probably have spent as long or longer if I was just doing this on some pleb distro because of build artefacts left over from successive attempts — and of course, most of the work would be rendered null next time I had to set up a new server! The amount of discovery (and number of dead ends) I got to rebase into concrete learnings is incredible.

(I once again express my thanks to those who got me here — especially @cadey@pony.social for putting the idea in my head years ago, and my ex-qpf for using Nix in the year of our lord 2023, which was a strong enough signal to finally Just Do It.)

Jambalam

2023-09-17T22:12:00Z

Have it your way. (Content note for just about everything.)

Further to fish fun in Nix refisited, today I wanted my git log to work a little different.

I have l as an alias for: (I’m going to add a bunch of newlines)

git log
  --show-notes='*'
  --abbrev-commit
  --pretty=format:'
    %Cred%h
    %Cgreen(%aD)%Creset
    -%C(bold red)%d%Creset
    %s
    %C(bold blue)<%an>%Creset
  '
  --graph

The output looks like this:

I’m focussing right at the end here: <%an> is what puts <Charlotte> at the end. an stands for “author name”; you can also do ae for “author email”, and cn and ce for committer name and email respectively.

Wouldn’t it be nice to see the author and committer here? Of course, if I just change it to something like <%an> <%cn>, we get this: (detacnurt for clarity)

…0) - (HEAD -> main, origin/main) 0004-happy-birthday: expand. <Charlotte> <Charlotte>
…0) - 0004-happy-birthday: publish. <Charlotte> <Charlotte>
…0) - TODOne. <Charlotte> <Charlotte>
…0) - Rules: relativize paths. <Charlotte> <Charlotte>
…) - TODO <Charlotte> <Charlotte>

In the majority of cases, author and committer are the same. GitHub does a neat thing where it shows the committer (or Co-authored-by:), but only if they’re different:

Unfortunately(?), git-log’s pretty formats don’t support “show this field only if it’s different to that one”. Maybe that’s a good thing. We’re here to break good things.

Here’s a patch to git that adds %cm and %cf. The naming semantics are “cute”: m and f are adjacent to n and e. The core addition to format_person_part (grumble) is as follows:

if (part == 'm' || part == 'f') {	 /* committer name or email if committer != author */
  if (ca_msg != NULL && !split_ident_line(&ca_s, ca_msg, ca_len)) {
    ca_name = ca_s.name_begin;
    ca_namelen = ca_s.name_end - ca_s.name_begin;
    ca_mail = ca_s.mail_begin;
    ca_maillen = ca_s.mail_end - ca_s.mail_begin;
    if (namelen == ca_namelen &&
        maillen == ca_maillen &&
        !strncmp(name, ca_name, namelen) &&
        !strncmp(mail, ca_mail, maillen))
      return placeholder_len;
  }
}

The outer conditional is hit for all %a and %c and looks for our m and f specifiers; the middle conditional ensures we’re in %c and that the Author (!) line can be parsed into name and mail address. Finally, we check that the Committer name and mail match the Author ones exactly. If so, we return immediately without appending anything to the output buffer. (We shortly thereafter treat m as identical to n and f as to e, in the case where they’re not matching.)

git show > pretty.c.patch and pkgs.git.overrideAttrs { patches = [./pretty.c.patch]; } gets you the goods. My alias now has <%an>% cm, meaning it’ll show the committer name (with a space) after the author name in gtlt, but only if it’s different. Here’s an example of that on Amaranth’s repository:

I really like this. In a tangible sense, my operating systems are becoming mine.

Happy birthday!

2023-08-22T05:24:00Z

I’m writing a little hardware I²C clock stretcher (I²C, oh! Big stretch) to help me make my I²C controller implementation actually support it.

These are some moments I’ve had while doing so.

Out-of-band experience

I added a tiny UART module to help me debug it. First I emitted a < character when starting a big stretch, and a > character once we’re all relaxed.

I was going to then write the number of cycles counted during the SCL tLOW period in decimal but eventually decided, Who Really Can Be Bothered, and just shoved it out onto the UART, LSB first.

The initial test went great:

@vyx ~> tio /dev/ttyUSB1 -b9600
[15:05:11.940] tio v2.5
[15:05:11.940] Press ctrl-t q to quit
[15:05:11.941] Connected
[15:10:05.065] Switched to hexadecimal mode
3c 0f 3e 3c 3e 3c 3e 3c 3e 3c 0f 3e

3c and 3e are < and > respectively. You can see I had some fun while realizing that I need to actually take steps to restart the FSM — disabling it isn’t enough.

The I²C bus is running at 400kHz, meaning we expect the SCL low period to last 1/800,000th of a second.

The iCEBreaker the sleepy kitty is running on is running at 12MHz. At that speed, 1/800,000th of a second passes in 12,000,000/800,000¹ = 15 cycles.

And we were seeing 0f in the output, 15! Perfect.

I recompiled the controller to run at 100kHz and continued the test.

3c 3c 3e 3c 3c 3e 3c 3c 3e 3c 3c 3e 3c 3c 3e 3c 3c 3e

??? I thought I made a logic error and we were somehow resetting back to the initial state without finishing measurement.

And then I said, “don’t fucking tell me,” because it’s not too hard to add 0f to itself repeatedly in your head and so 0f, 1e, 2d, 3c. Happy birthday!

At this point I promptly changed the start/stop characters to ff and fe, and then — detecting that I was just setting up the next, much larger footgun for myself — decided to dump the count one nibble at a time and thus render any byte with a non-zero high nibble officially out-of-band, and thus:

ff 0c 03 fe

Modes of measurement

The I²C controller I’m testing with/for outputs its clock at a 50% duty cycle exactly². I probably even verified that with a logic analyzer or oscilloscope at some stage! Point is, my initial idea was to train on the SCL tLOW period, and then start holding it low for ~twice that period as of the next low, and thus stretch the clock (a LOT).

When you’re a noob like me, you may encounter this:

Here we have the bus only getting pulled up halfway. This is what it looks like when someone is trying to ground your bus at the same time as someone trying to put high out on it. An I²C bus is designed to be pulled high so that anyone can pull it low, but we’re seeing it driven high.

The controller needed to turn SCL’s output-enable off, to let it get pulled up high on its own, and then only to switch the output-enable on when it needs to be driven low. This lets anyone else on the bus keep SCL low, thus stretching the clock.

This worked nicely when it comes to letting the bus stretch, but today I was trying to get the measurement to come out right — I wanted to have the little debugging app that talks via UART reporting the correct I²C bus speed — and noticed that, actually, we’re getting it wrong not only because I’m failing to count correctly, but because the waveform doesn’t have a 50% duty cycle!

My poor baby oscilloscope can’t actually measure fast enough to see it, but turns out letting a pull-up resistor bring a bus high takes a little bit longer than driving it high. As a result, the tail end of the tLOW period is eating into the start of tHIGH as it shakily makes its way back up to full voltage. I absolutely need a way to see this, so I need a better scope I guess.

Anyway, I’ll also measure from rising edge to rising edge (and falling edge to falling edge), and that should give me some more insight. Logically, I expect the falling-to-falling to be very consistent, because that transition is driven, whereas the rising-to-rising might vary depending on when the signal gets high enough to be considered “high” each cycle.

I just need to not accidentally reinvent the Glasgow Interface Explorer while I’m here.

Footnotes

1/(12MHz/800kHz) = 1/((1/12,000,000)/(1/800,000)) = 12,000,000/800,000. ↩
Hey, that sounds like something I could really formally verify. I have the start of a verification setup, might as well use it. ↩

Nix revisited

2023-07-14T16:42:00Z

I realized I was in error in not using Nix, and have been addressing that. (The primary artifact so far that is public is hdx, a response to Installing an HDL toolchain from source.) I have some knowledge of it from previous experiments. Some observations:

You must thread the needle between “properly sitting down and reading the language guide” and “actively replacing previously-statefully-configured parts of your system and build environments”. Without the former none of the idioms make sense; without the latter you won’t remember anything from the former.
If you’re fighting Nix, you’re probably missing a good opportunity to use it instead. Here’s an example:
1. As a heavy Git user, I have a lot of terse aliases which are part of my muscle memory.
2. It’s unacceptable to me to type git or even g before those aliases, as even the latter represents a 200% additional load on commands I use extremely frequently.
3. It’s preferable to me to use Git’s aliases over shell aliases to do the actual expansion, particularly as I use some, uh, “Git shell aliases”?
I used to use a method that involved piping the output of this Ruby script into source in my shell rc:
```
#!/usr/bin/env ruby

alias_lines = `git config --global --list`.lines.grep(/^alias\./)

alias_lines.each do |line|
  line =~ /\Aalias\.([^=]+)=(.*)\n\z/
  name, exp = $1, $2

  if exp =~ /\A!/
    puts "alias #{name}=\"git #{name}\"  # #{exp}"
  else
    puts "alias #{name}=\"git #{exp}\""
  end
end
```
(I think the conditional was trying to make up for lack of completions through Git aliases, which isn’t necessary these days.)

I almost took that with me. Can you believe it? I now have a gitAliases.nix that looks like the following¹:
```
{
  co = "checkout";
  cb = "checkout -b";
  pc = "checkout -p";

  s = "status -sb";

  b = "branch";
  ba = "branch -a";
  bd = "branch -d";

# ...
```
Then, in my Home Manager configuration, effectively the following:
```
let
  gitAliases = import ./gitAliases.nix;
in
{
  home-manager.users.charlotte = {
    programs.fish = {
      shellAliases = {
        # ...
      } // builtins.mapAttrs (name: _v: "git ${name}") gitAliases;
    };

    programs.git = {
      aliases = gitAliases;
    };
  };
}
```
This feels delicious. I have just noticed I’m not getting --wraps set when defining the function, and so completions are not provided. It appears slightly unpredictable on fish’s side whether alias x y will use -w y, possibly to do with when in init it’s happening and whether such a function has been defined before. Oh, here we go.
- Time to hack that apart. It’s almost disgusting how easy Nix makes patching packages I use and then having that just appear on all my systems! Fuck! I’m sure there’s a less nuclear option but I just wanna.
  - Edit: this is the stuff dreams are made of.
As with anything, keep the stdlib source open in a window/tab/pane. Here this means /nix/var/nix/profiles/per-user/root/channels/nixpkgs/.

Builds may not generally be reproducible between NixOS and Nix on a different platform. Ahem.

I think this implies building Nixpkgs 23.05’s icestorm on macOS today would fail.

Let’s verify. We want to use nix-build --option substitute false to disable binary substitution, but first invoke the nix-shell once for the derivation so we don’t build all its dependencies from source too:

~ $ # After a lot of fucking around with nix-store --gc:
~ $ nix-shell '<nixpkgs>' -A icestorm
these 16 paths will be fetched (2.62 MiB download, 16.98 MiB unpacked):
  /nix/store/sm3f0jqk0y1bmwpprjy15icb7bw9kfyp-apple-framework-CoreFoundation-11.0.0
  /nix/store/iqh2hzmrnj9rvw6ahdzzsp9cqzf3ji6w-cctools-binutils-darwin-wrapper-973.0.1

[ ... lots of output ... ]

copying path '/nix/store/7v4rbxd8i0hsk2hgy8jnd4qn9vk89a86-clang-wrapper-11.1.0' from 'https://un5nfj1egjpbbbnrx28f6wr.julianrbryant.com'...
copying path '/nix/store/mas4ifv1v6llnqkyxq5w235x0hdq5yq3-stdenv-darwin' from 'https://un5nfj1egjpbbbnrx28f6wr.julianrbryant.com'...

[nix-shell:~]$ exit
exit
~ $ nix-build --option substitute false '<nixpkgs>' -A icestorm
this derivation will be built:
  /nix/store/iqw5iqqkm71vx5dl4s6xzpm5ymxjddyq-icestorm-2020.12.04.drv
building '/nix/store/iqw5iqqkm71vx5dl4s6xzpm5ymxjddyq-icestorm-2020.12.04.drv'...

[ ... lots of output ... ]

patching script interpreter paths in /nix/store/jxfzqadgp6ygd0dfdi7s0jx0nwbd3kxh-icestorm-2020.12.04
stripping (with command strip and flags -S) in  /nix/store/jxfzqadgp6ygd0dfdi7s0jx0nwbd3kxh-icestorm-2020.12.04/bin
/nix/store/jxfzqadgp6ygd0dfdi7s0jx0nwbd3kxh-icestorm-2020.12.04
~ $

It works! Joke’s on me: the revision used in Nixpkgs is about 8 commits before the macOS fix that now needs to be worked around.

Let’s verify this by building the derivation with the revision overridden. The name is also overridden, to avoid the package name + version being used:

with import <nixpkgs> {};
icestorm.overrideAttrs ({ src, ... }: {
  name = "icestorm";
  src = src.override {
    rev = "d20a5e9001f46262bf0cef220f1a6943946e421d";
    sha256 = lib.fakeSha256;
  };
})

We do the little dance to get the fixed-output derivation hash suitable for the site it’s used:

~ $ nix-build -E 'with import <nixpkgs> {}; icestorm.overrideAttrs ({ src, ... }: { name = "icestorm"; src = src.override { rev = "d20a5e9001f46262bf0cef220f1a6943946e421d"; sha256 = lib.fakeSha256; }; })'
these 2 derivations will be built:
  /nix/store/wpcxl7fz89sk1b45xy2m36cv3gljgzmp-source.drv
  /nix/store/gjs2rm2la5as2yh139yaqcz0q5hjgsc7-icestorm.drv
building '/nix/store/wpcxl7fz89sk1b45xy2m36cv3gljgzmp-source.drv'...

trying https://github.com/YosysHQ/icestorm/archive/d20a5e9001f46262bf0cef220f1a6943946e421d.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  926k    0  926k    0     0  1195k      0 --:--:-- --:--:-- --:--:-- 5570k
unpacking source archive /private/tmp/nix-build-source.drv-0/d20a5e9001f46262bf0cef220f1a6943946e421d.tar.gz
error: hash mismatch in fixed-output derivation '/nix/store/wpcxl7fz89sk1b45xy2m36cv3gljgzmp-source.drv':
         specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
            got:    sha256-dEBmxO2+Rf/UVyxDlDdJGFAeI4cu1wTCbneo5I4gFG0=
error: 1 dependencies of derivation '/nix/store/gjs2rm2la5as2yh139yaqcz0q5hjgsc7-icestorm.drv' failed to build
~ $ nix-build -E 'with import <nixpkgs> {}; icestorm.overrideAttrs ({ src, ... }: { name = "icestorm"; src = src.override { rev = "d20a5e9001f46262bf0cef220f1a6943946e421d"; sha256 = "dEBmxO2+Rf/UVyxDlDdJGFAeI4cu1wTCbneo5I4gFG0="; }; })'
this derivation will be built:
  /nix/store/9klhbj6biiqn9696zzvyy8cmyjfjaw2d-icestorm.drv
building '/nix/store/9klhbj6biiqn9696zzvyy8cmyjfjaw2d-icestorm.drv'...

[ ... lots of output ... ]

       > cp icebox_vlog.py    /nix/store/1rdvdwvz44kkhirbvpn0yx2njwalrbf2-icestorm/bin/icebox_vlog
       > cp icebox_stat.py    /nix/store/1rdvdwvz44kkhirbvpn0yx2njwalrbf2-icestorm/bin/icebox_stat
       > sed -i '' 's+import iceboxdb+import iceboxdb as iceboxdb+g' /nix/store/1rdvdwvz44kkhirbvpn0yx2njwalrbf2-icestorm/bin/icebox.py
       > sed: can't read s+import iceboxdb+import iceboxdb as iceboxdb+g: No such file or directory
       > make[1]: *** [Makefile:65: install] Error 2
       > make[1]: Leaving directory '/private/tmp/nix-build-icestorm.drv-0/source/icebox'
       > make: *** [Makefile:13: install] Error 2
       For full logs, run 'nix-store -l /nix/store/9klhbj6biiqn9696zzvyy8cmyjfjaw2d-icestorm.drv'.
~ $

Sure enough, it does fail.

Nix feels very appropriate for people like me, whose thought processes or short-term memory may be disturbed without warning, thanks to the nature of work-in-progress state with declarative systems.
- By which I mean; for the most part, recovering the idea I’m halfway through an attempt of² is more “image load” than “procedural init”. It doesn’t require parsing shell history or terminal scrollback in order to learn the meaning of the current state of my system—99% of the context is in a file.
I’m avoiding nix-env and flakes. I don’t like the look of workflows that involve either. Xe describes flakes as being suitable for use cases where you might use Niv or Lorri. Niv and Lorri also appear to be tools for workflows I don’t like the look of. Lorri refers to “fast direnv integration for robust CLI and editor integration”, and for whatever reason, that’s a slightly repellent notion to me at this stage.

I expect my opinion on flakes will change as I continue.

Did I mention builds may not be reproducible?

-- Build files have been written to: /tmp/nix-build-nextpnr.drv-1/nextpnr-54b2045/build
building
build flags: -j10 SHELL=/nix/store/mxvgjwzdvrl81plvgqnzbrqb14ccnji6-bash-5.2-p15/bin/bash
[  0%] Building CXX object bba/CMakeFiles/bbasm.dir/main.cc.o
[  1%] Generating chipdb/chipdb-25k.bba
/nix/store/mxvgjwzdvrl81plvgqnzbrqb14ccnji6-bash-5.2-p15/bin/bash: line 1: 54809 Segmentation fault: 11  /nix/store/zdd58zb8y7bm15jm0985fdjzy8wrmaci-python3-3.11.4/bin/python3.11 /tmp/nix-build-nextpnr.drv-1/nextpnr-54b2045/ecp5/trellis_import.py -L /nix/store/z3mpz8mqd858vbx849zqyh1mdv64l3vd-trellis/lib/trellis -L /nix/store/z3mpz8mqd858vbx849zqyh1mdv64l3vd-trellis/share/trellis/util/common -L /nix/store/z3mpz8mqd858vbx849zqyh1mdv64l3vd-trellis/share/trellis/timing/util -p /tmp/nix-build-nextpnr.drv-1/nextpnr-54b2045/ecp5/constids.inc -g /tmp/nix-build-nextpnr.drv-1/nextpnr-54b2045/ecp5/gfx.h 25k > chipdb/chipdb-25k.bba.new
make[2]: *** [ecp5/CMakeFiles/chipdb-ecp5-bbas.dir/build.make:77: ecp5/chipdb/chipdb-25k.bba] Error 139
make[1]: *** [CMakeFiles/Makefile2:359: ecp5/CMakeFiles/chipdb-ecp5-bbas.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[  2%] Linking CXX executable bbasm
[  2%] Built target bbasm
make: *** [Makefile:136: all] Error 2
error: boost::bad_format_string: format-string is ill-formed
~ $ # ...

(There’s a segmentation fault in there :)))

Footnotes

If my internal sense of what a Git alias should be called is occupied by a base system command that itself is in muscle memory—which only occurs for 2-letter aliases—I transpose the two letters, or repeat a letter somewhere.
- git checkout -p thus becomes pc to avoid cp(1).
- git cherry-pick is uncommon enough that it loses the fight for pc and gets pcp.
- git rm gets mrm, because mr on its own feels like it should be obviously merge-related — there are 9 aliases beginning with m that are merge-related — but at 3 characters, mrm is unique enough to be recognizable.
- Why not cpc or rmr? iirc, checkout -p got pc first; when it was time to introduce a cherry-pick alias, there was no consideration of giving it cpc—pc was an established metaphor for this initialism, whereas cpc would break that and introduce confusion. Moving checkout -p to cpc for consistency’s sake is unacceptable and leaves no clear answer for cherry-pick. rmr seems fine, but continuing with the weirdness is what makes a beautiful natural language :)
↩
Which I have to do once every 8–10 minutes on average, at a guess. ↩

Untangling cycles

2023-06-29T00:51:00Z

This is straight from my journal, so it starts without warning.

The bit packing is turning out to be surprisingly tricky!

Memory is synchronous but our uses of addr[0] were all comb, so they didn’t align with the actual target in the cycle it got transmitted from memory when we were advancing addr every cycle. This was a really good exercise in Being Confused As Heck.

Going to try to explicate the above a bit more clearly for my own elucidation. Ignoring the write half of the equation for simplicity—the issues faced are the same.

This post is literate Python. Why not. We have the following as baseline:

import math
from typing import Optional

from amaranth import Elaboratable, Memory, Module, Record, Signal
from amaranth.build import Platform
from amaranth.hdl.ast import ShapeCastable
from amaranth.hdl.mem import ReadPort
from amaranth.hdl.rec import DIR_FANIN, DIR_FANOUT
from amaranth.sim import Simulator


class ROMBus(Record):
    def __init__(self, addr: ShapeCastable, data: ShapeCastable):
        super().__init__(
            [
                ("addr", addr, DIR_FANIN),
                ("data", data, DIR_FANOUT),
            ],
            name="ROMBus",
        )

class Downstream(Record):
    def __init__(self):
        super().__init__(
            [
                ("data", 8, DIR_FANIN),
                ("stb", 1, DIR_FANIN),
            ]
        )

A ROMBus is a connectable path to access some read-only memory. Downstream here is a hypothetical recipient of data being read from ROM. (The ROM is actually a RAM that gets filled on power-on from flash.)

The key problem I was solving was that, until now, I’ve been storing all my data in 8-bit wide Memory instances, but a lot of the actual embedded block RAM I’m using has 16-bit wide words. As a result, the upper 8 bits of every word has been left unused.

It’d be nice to add a translation layer that transparently forwarded reads and writes from an 8-bit addressable space into the 16-bit words. Even bytes in the lower halves, odd bytes in the upper halves. Here’s what that’d look like:

ROM_CONTENT_PACKED = [0x2211, 0x4433, 0x6655, 0x8877]
ROM_LENGTH = 8

The length of the ROM that all the downstream consumers care about is the 8-bit addressable one—address 0 has 0x11, address 1 0x22, etc. The fact that we have 8 bytes packed into 4 words of 16 bits is irrelevant to them.

Here’s where our example will play out:

class Example(Elaboratable):
    def __init__(self):
        self.downstream = Downstream()

    def elaborate(self, platform: Optional[Platform]):
        m = Module()

The Downstream is exposed on the instance so we can access it from our simulator process.

We now need to do the following things:

Determine the size of the packed RAM.
Create our Memory instance for it.
- We’ll initialize it with init here, and completely ignore the write aspect of the scenario. The issues it will suffer from are the same. (I sure suffered!)
Get the ReadPort for our RAM. I’m asserting the lengths here illustratively for the reader’s benefit.

        packed_size = math.ceil(ROM_LENGTH / 2)
        rom_mem = Memory(
            width=16,
            depth=packed_size,
            init=ROM_CONTENT_PACKED,
        )
        m.submodules.rom_rd = rom_rd = rom_mem.read_port()
        assert len(rom_rd.addr) == 2
        assert len(rom_rd.data) == 16

rom_rd.addr determines the address in the 16-bit-wide RAM (0x0–0x3), and rom_rd.data returns those 16 bits. Memory is synchronous by default (and the read enable is also always on under default settings), so, given a made-up mem[x] operator, the following timeline applies:

cycle n+0: some process assigns rom_rd.addr.eq(x)
cycle n+1: the read port sees its new addr value and assigns rom_rd.data.eq(mem[x])
cycle n+2: rom_rd.data takes the value of mem[x]

Now we’ll create our ROMBus. This is what all the RTL I had was already using—it was connected directly to the read port of the 8-wide memory.

        rom_bus = ROMBus(range(ROM_LENGTH), 8)
        assert len(rom_bus.addr) == 3
        assert len(rom_bus.data) == 8

We’re going to put the actual translation logic and state machine in separate functions, so they can be changed later while preserving the literacy of this post. Why not.

        self.translation(m, rom_rd, rom_bus)
        self.fsm(m, rom_bus)

        return m

We want to hook up the ROM bus to the memory in a transparent fashion. Here’s what I started with:

    def translation(self, m: Module, rom_rd: ReadPort, rom_bus: ROMBus):
        m.d.comb += [
            rom_rd.addr.eq(rom_bus.addr >> 1),
            rom_bus.data.eq(
                rom_rd.data.word_select(rom_bus.addr[0], 8)
            ),
        ]

We shift off the last bit of the input (8-bit) address to create the output (16-bit) address, creating the following mapping:

8-bit address	16-bit address
`0x0` / `0b000`	`0x0` / `0b00`
`0x1` / `0b001`	`0x0` / `0b00`
`0x2` / `0b010`	`0x1` / `0b01`
`0x3` / `0b011`	`0x1` / `0b01`
`0x4` / `0b100`	`0x2` / `0b10`
`0x5` / `0b101`	`0x2` / `0b10`
`0x6` / `0b110`	`0x3` / `0b11`
`0x7` / `0b111`	`0x3` / `0b11`

We select the 8-bit word from the 16-bit data coming out of the memory corresponding to the LSB of the input (8-bit) address.
- a.word_select(b, w) is essentially a[b*w : (b+1)*w].
- When the LSB of the 8-bit address is 0, this will select rd_data[0:8]. When the LSB is 1, this will select rd_data[8:16].
- So:
  - 8-bit address 0x0 will select mem[0x0][0:8],
  - 8-bit address 0x1 will select mem[0x0][8:16],
  - 8-bit address 0x2 will select mem[0x1][0:8],
  - 8-bit address 0x3 will select mem[0x1][8:16],
  - etc.

Now we implement a reader from our ROM:

    def fsm(self, m: Module, rom_bus: ROMBus):
        m.d.sync += self.downstream.stb.eq(0)

        with m.FSM():
            with m.State("INITIAL"):
                # cycle n+0
                m.d.sync += rom_bus.addr.eq(0)
                m.next = "WAIT"

            with m.State("WAIT"):
                # cycle n+1 / n'+1
                m.next = "READ"

            with m.State("READ"):
                # cycle n+2, n'+0
                m.d.sync += [
                    self.downstream.data.eq(rom_bus.data),
                    self.downstream.stb.eq(1),
                    rom_bus.addr.eq(rom_bus.addr + 1),
                ]
                m.next = "WAIT"

This is a simple process that reads data and passes them along to some downstream process (which needs to be able to accept this data as fast as we give it to them!).

We start at address zero (n+0),
wait a cycle for the memory to see it (n+1),
and then pass it to the downstream (n+2) while advancing the address we read (n’+0).
The next cycle we’re back in WAIT as advanced address is seen by the memory (n’+1).

We end up strobing the downstream every other cycle. (That strobe is seen in the n+1 / n’+1 cycle.)

Let’s simulate it and report the results:

def main():
    dut = Example()

    def process():
        count = 0
        yield
        while True:
            if (yield dut.downstream.stb):
                print(f"data: {(yield dut.downstream.data):02x}")
                count += 1
                if count == 8:
                    return
            yield

    sim = Simulator(dut)
    sim.add_clock(1e-6)
    sim.add_sync_process(process)
    sim.run()

This can now be run:

$ python -c 'import ex; ex.main()'
data: 11
data: 22
data: 33
data: 44
data: 55
data: 66
data: 77
data: 88

It’s perfect!

Almost. Let’s revisit the timeline for accessing the synchronous memory:

cycle n+0: rom_rd.addr.eq(x)
cycle n+1: read port sees new addr, assigns rom_rd.data.eq(mem[x])
cycle n+2: rom_rd.data sees mem[x]

The important part is that you can assign a new address y in cycle n+1, without impacting what happens in cycle n+2, such that mem[y] is now available to use in cycle n+3. The read port will only see the address y in the same cycle that it’s already propagated mem[x] into its data register.

Let’s now change our state machine to take advantage of this:

def fsm(self: Example, m: Module, rom_bus: ROMBus):
    m.d.sync += self.downstream.stb.eq(0)

    with m.FSM():
        with m.State("INITIAL"):
            # cycle n+0
            m.d.sync += rom_bus.addr.eq(0)
            m.next = "WAIT"

        with m.State("WAIT"):
            # cycle n+1, n'+0
            m.d.sync += rom_bus.addr.eq(1)
            m.next = "READ"

        with m.State("READ"):
            # cycle n+2, n'+1, n''+0
            m.d.sync += [
                self.downstream.data.eq(rom_bus.data),
                self.downstream.stb.eq(1),
                rom_bus.addr.eq(rom_bus.addr + 1),
            ]


Example.fsm = fsm

We start at address zero (n+0),
while waiting a cycle for the memory to see it (n+1), we also increment the address to one (n’+0),
and then pass the first result the downstream (n+2), while the memory is just now seeing the second result (n’+1), and simultaneously increment the address we read (n’’+0).

We don’t change state once we’re in READ: every cycle we hand to downstream the data from the address we set two cycles ago; every cycle the memory is seeing the address we gave one cycle ago; every cycle we increment the address to keep it going.

(My wording here muddles up the timing of when we “set” a given value quite a lot — really, we initiate the setting of the address two cycles ago, which one cycle ago is set (and seen), which this cycle we then see the data returned of.)

This is pretty theoretical in this form, but I have a few state machines that do this kind of sliding continuous read in a limited fashion.

So what happens?

$ python -c 'import ex; ex.main()'
data: 22
data: 11
data: 44
data: 33
data: 66
data: 55
data: 88
data: 77

All the bytes are reversed! (This was a lot weirder to debug when the same problem might have been affecting the initial write to RAM, too.)

Why?

We’ll review the translation statements:

m.d.comb += [
    rom_rd.addr.eq(rom_bus.addr >> 1),
    rom_bus.data.eq(
        rom_rd.data.word_select(rom_bus.addr[0], 8)
    ),
]

This translation happens in the combinatorial domain, meaning that rom_rd.addr will change to rom_bus.addr >> 1 as soon as a change on rom_bus.addr is registered — there isn’t an additional cycle between the requested 8-bit address on the ROM bus changing and the read port’s 16-bit address changing:

cycle	statement issued	ROM bus addr	read port addr	read port data
0	`rom_bus.addr.eq(0)`	x	x	x
1	`rom_bus.addr.eq(1)`	`0`	`0`	x
2	`rom_bus.addr.eq(2)`	`1`	`0`	`0x2211`
3	`rom_bus.addr.eq(3)`	`2`	`1`	`0x2211`
4	`rom_bus.addr.eq(4)`	`3`	`1`	`0x4433`
5	`rom_bus.addr.eq(5)`	`4`	`2`	`0x4433`

Similarly, the ROM bus data port will be updated as soon as the read port’s data port (rom_rd.data) changes.

It will also be updated as soon as the LSB of the ROM bus’s requested address changes (rom_bus.addr[0]).

But by the time we’re actually getting data in the read port for an address, the ROM bus has registered the next address! Thus we select the half of the 16-bit word based on the LSB of the following address, which (given the addresses are sequential) will always be the opposite half to the one we really want:

cycle	ROM bus addr	read port data	ROM bus addr [0]	ROM bus data
0	x	x	x	x
1	`0`	x	`0`	x
2	`1`	`0x2211`	`1`	`0x22`
3	`2`	`0x2211`	`0`	`0x11`
4	`3`	`0x4433`	`1`	`0x44`
5	`4`	`0x4433`	`0`	`0x33`

We need to introduce a delay in the address as used by the translation on the way back out, to account for the fact that read data corresponds to the address from the previous registered cycle, not this one:

def translation(
    self: Example,
    m: Module,
    rom_rd: ReadPort,
    rom_bus: ROMBus,
):
    last_addr = Signal.like(rom_bus.addr)
    m.d.sync += last_addr.eq(rom_bus.addr)

    m.d.comb += [
        rom_rd.addr.eq(rom_bus.addr >> 1),
        rom_bus.data.eq(rom_rd.data.word_select(last_addr[0], 8)),
    ]


Example.translation = translation

This gives:

cycle	ROM bus addr	last ROM bus addr	read port data	last ROM bus addr [0]	ROM bus data
0	x	x	x	x	x
1	`0`	x	x	x	x
2	`1`	`0`	`0x2211`	`0`	`0x11`
3	`2`	`1`	`0x2211`	`1`	`0x22`
4	`3`	`2`	`0x4433`	`0`	`0x33`
5	`4`	`3`	`0x4433`	`1`	`0x44`

And so:

$ python -c 'import ex; ex.main()'
data: 11
data: 22
data: 33
data: 44
data: 55
data: 66
data: 77
data: 88

I like how the x’s in this table don’t flow back “up” in time as the data dependencies flow right, whereas in the previous table, they do.

Installing an HDL toolchain from source

2023-06-27T09:12:00Z

It occurred to me while writing up § Requirements in the README for some¹ gateware that getting the whole beginner’s open-source FPGA toolchain set up can be a serious stumbling block, as I imagine it probably was for me once.

The main pre-packaged solution that comes to mind is OSS CAD Suite. It’s excellent, but common to “all-in-one” solutions, it makes assumptions that mean it’s prone to inflexibility in certain ways. Rebuilding just one of the tools is not always as simple as that — Python environment or shared library conflicts can result, and the tools are wrapped so as to always prefer their own distribution, necessitating further hacks for those that call each other.

With any luck, you won’t run into any of these cases, but if you do, getting everything built for yourself correctly can be a bit vexing — the YosysHQ tools’ documentation tends to point back to their own pre-built packages (and products) in preference to (and sometimes instead of) instructing on how to build.

I’ve done this process three times recently while rejigging my development environments, so I’m describing it for posterity/others.

Scope

By the end of this guide, you’ll have the following ready to go:

Git is for acquiring source code.

Python 3 is for using Amaranth.

Amaranth is the HDL I’m using. It is a Python library which consists of a language for describing digital logic, as well as facilitating simulation and building of the resulting designs. It integrates well with the ecosystem, and permits intermixing with Verilog (or VHDL). At time of writing, its development pace has quickened.

Yosys is the synthesis framework at the heart of Amaranth. It is a digital logic synthesizer, which is a phrasing that severely understates how much work is involved—for more information, see the Yosys manual.

Amaranth actually comes with its own portable Yosys built-in, which works beautifully. We’ll use it, but we’ll build it separately, too: for cases when we want to make our own changes, or use step-through debugging to understand what’s happening. It’s also necessary for formal verification.

nextpnr and Project IceStorm are for targeting the Lattice iCE40 family of FPGAs, which is known for its relative accessibility. I’ve been learning with an iCEBreaker (see also), which is built around the iCE40UP5k FPGA, and have found this to be true.

SymbiYosys and Z3 are for formal verification. I promise it’s good.

Instructions are verified for Linux x86_64 and macOS arm64. I intended to cover Windows, too, but over four months found the experience inconsistent² enough that it was easier to use WSL 2³. On Linux and WSL, I’ve used Debian.

I assume Linux users can install packages using the distribution package manager, and macOS users using Homebrew. I’m going to avoid installing almost anything globally, however, that wouldn’t already get installed by your package manager as a matter of course, especially when there’s reasons you might need multiple versions around.

Git

The Git website’s Downloads page has instructions for acquiring it through your relevant package manager.

Python 3

That was easy. Now the opinions start.

Install the asdf Multiple Version Runtime Manager. The Getting Started page has commands for dependency installation through package manager. Use the official git method to download asdf itself, and then follow the instructions for your shell.

Now install the Python asdf plugin, and install the latest stable version of Python:

~ $ asdf plugin add python
initializing plugin repository...Cloning into '/home/charlotte/.asdf/repository'...
remote: Enumerating objects: 5273, done.
remote: Counting objects: 100% (481/481), done.
remote: Compressing objects: 100% (88/88), done.
remote: Total 5273 (delta 419), reused 445 (delta 393), pack-reused 4792
Receiving objects: 100% (5273/5273), 1.21 MiB | 29.47 MiB/s, done.
Resolving deltas: 100% (2849/2849), done.
~ $ asdf latest python
3.11.4
~ $ asdf install python 3.11.4
python-build 3.11.4 /home/charlotte/.asdf/installs/python/3.11.4
Downloading Python-3.11.4.tar.xz...
-> https://un5gmtkzgjcywhd1hkae4.julianrbryant.com/ftp/python/3.11.4/Python-3.11.4.tar.xz
Installing Python-3.11.4...
Installed Python-3.11.4 to /home/charlotte/.asdf/installs/python/3.11.4
~ $ asdf global python 3.11.4
~ $

(You might get some warnings about extensions not being compiled. That’s OK. There’s also 3.12.0b3 available at time of writing, if you don’t mind a beta.)

The last command makes it the default Python for our user. asdf puts some shims in our PATH which use a combination of our configured defaults (global), our current path (local), and environment variables (shell) to select the desired version:

~ $ which python
/home/charlotte/.asdf/shims/python
~ $ asdf current python
python          3.11.4          /home/charlotte/.tool-versions
~ $ asdf where python
/home/charlotte/.asdf/installs/python/3.11.4
~ $ asdf which python
/home/charlotte/.asdf/installs/python/3.11.4/bin/python
~ $ python
Python 3.11.4 (main, Jun 26 2023, 16:06:57) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

venv

The last thing we want to do is actually a per-project step. We’re about to install Amaranth, which is a Python dependency, and so we want to make sure we’re installing Python dependencies in a separate virtual environment per project, that they don’t interfere or conflict with each other.

In your project directory, create a new virtual environment called venv, and then activate it:

prj $ python -m venv venv
prj $ source venv/bin/activate
(venv) prj $

(Note there are a few different activate variants in the bin directory for different shells.)

Add venv to your .gitignore or similar.

It’s important to remember to activate the virtual environment before running Python or installing dependencies with pip. Many IDEs will automatically activate (or prompt to activate) virtual environments when they’re detected in the root of a project. Similarly, some shells can be configured to do similar.

Note that the Python instance used by the virtual environment is tied to the specific version we had chosen through asdf, and not the shim:

(venv) prj $ readlink venv/bin/python
/home/charlotte/.asdf/installs/python/3.11.4/bin/python
(venv) prj $

We’re ready to install Python dependencies.

Amaranth

Firstly, note Amaranth’s own installation instructions. We’ll follow along, and deviate from them somewhat.

Install GTKWave from your package manager. We’ll come back to Yosys.

Verify we do in fact have the latest pip:

(venv) prj $ pip install --upgrade pip
Requirement already satisfied: pip in ./venv/lib/python3.11/site-packages (23.1.2)
(venv) prj $

We do not pass --user to pip—it is rejected in a virtual environment, for --user implies writing to your home directory, which would escape the virtual environment.

We’re going to skip the latest release and go straight to an editable development snapshot. You may want to clone it within your project directory, perhaps as a Git submodule, or along-side. I’m going with along-side.

Clone Amaranth and install it in editable mode, with the built-in Yosys:

(venv) prj $ cd ..
(venv) ~ $ git clone https://github.com/amaranth-lang/amaranth
Cloning into 'amaranth'...
remote: Enumerating objects: 8651, done.
remote: Counting objects: 100% (272/272), done.
remote: Compressing objects: 100% (95/95), done.
remote: Total 8651 (delta 170), reused 227 (delta 162), pack-reused 8379
Receiving objects: 100% (8651/8651), 1.71 MiB | 29.14 MiB/s, done.
Resolving deltas: 100% (6474/6474), done.
(venv) ~ $ cd amaranth
(venv) amaranth $ pip install --editable .[builtin-yosys]
Obtaining file:///home/charlotte/amaranth
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done

[... lots of output ...]

Successfully built amaranth
Installing collected packages: wasmtime, pyvcd, MarkupSafe, Jinja2, amaranth-yosys, amaranth
Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.3 amaranth-0.4.dev134+g99417d6 amaranth-yosys-0.25.0.0.post72 pyvcd-0.4.0 wasmtime-9.0.0
(venv) amaranth $

Note that the virtual environment remained active even as we left the directory we created it in. This is desirable: it means the editable snapshot was installed in our virtual environment.

We’ll install the board definitions now, too. Clone the repository and install it the same way, except without the [builtin-yosys] option:

(venv) amaranth $ cd ..
(venv) ~ $ git clone https://github.com/amaranth-lang/amaranth-boards
Cloning into 'amaranth-boards'...
remote: Enumerating objects: 1353, done.
remote: Counting objects: 100% (532/532), done.
remote: Compressing objects: 100% (136/136), done.
remote: Total 1353 (delta 426), reused 405 (delta 396), pack-reused 821
Receiving objects: 100% (1353/1353), 307.90 KiB | 17.11 MiB/s, done.
Resolving deltas: 100% (943/943), done.
(venv) ~ $ cd amaranth-boards/
(venv) amaranth-boards $ pip install --editable .
Obtaining file:///home/charlotte/amaranth-boards
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done

[... lots of output ...]

Successfully built amaranth-boards
Installing collected packages: amaranth-boards
Successfully installed amaranth-boards-0.1.dev228+g54e6ac4
(venv) amaranth-boards $

We’re ready. We can verify the installations by using a Python shell in the virtual environment:

(venv) prj $ python
Python 3.11.4 (main, Jun 26 2023, 16:06:57) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from amaranth import *
>>> Signal
<class 'amaranth.hdl.ast.Signal'>
>>> from amaranth_boards.icebreaker import *
>>> ICEBreakerPlatform
<class 'amaranth_boards.icebreaker.ICEBreakerPlatform'>
>>>

Yosys

Clone the Yosys repo and read its README. § Building from Source tells you which packages from the package manager it needs.

By default, Yosys will install into /usr/local, but we’ll override it to use ~/.local instead. We do this by setting the PREFIX variable, and importantly, we need it set for the build step too, not only install. Otherwise, the yosys-config helper that gets installed will report the wrong values.

Yosys’s Makefile will include Makefile.conf if it exists; we’ll put it in there so we can’t forget, and don’t have to stash Makefile changes when we pull the latest. Then we build in parallel and install:

(venv) yosys $ echo 'PREFIX = $(HOME)/.local' > Makefile.conf
(venv) yosys $ make -j8
[Makefile.conf] PREFIX = $(HOME)/.local
[  0%] Building kernel/version_2310a0ea9.cc
[  0%] Building kernel/driver.o
[  0%] Building techlibs/common/simlib_help.inc
[  0%] Building techlibs/common/simcells_help.inc
[  1%] Building kernel/rtlil.o

[... lots of output ...]

[ 94%] ABC: `` Compiling: /src/bdd/llb/llb4Nonlin.c
[ 94%] ABC: `` Compiling: /src/bdd/llb/llb4Sweep.c
[ 94%] ABC: `` Building binary: abc-1de4eaf
[100%] Building yosys-abc

  Build successful.

(venv) yosys $ make install
[Makefile.conf] PREFIX = $(HOME)/.local
mkdir -p /home/charlotte/.local/bin
cp yosys yosys-config yosys-abc yosys-filterlib yosys-smtbmc yosys-witness /home/charlotte/.local/bin
strip -S /home/charlotte/.local/bin/yosys
strip /home/charlotte/.local/bin/yosys-abc
strip /home/charlotte/.local/bin/yosys-filterlib
mkdir -p /home/charlotte/.local/share/yosys
cp -r share/. /home/charlotte/.local/share/yosys/.
(venv) yosys $

You may need to add ~/.local/bin to your PATH. Test the installed binary. Check the yosys-config output:

(venv) yosys $ yosys-config --datdir
/home/charlotte/.local/share/yosys
(venv) yosys $

Project IceStorm

Before nextpnr, we need the technology-specific support. That’s this step.

Clone Project IceStorm. There’s no Makefile.conf here, so edit the first line of config.mk:

-PREFIX ?= /usr/local
+PREFIX = $(HOME)/.local

Install the libftdi development package; it’s libftdi-dev on Debian and libftdi in Homebrew.

Now compile and install Project IceStorm. I’ve avoided compiling in parallel as its build script sometimes gets ahead of itself:

(venv) icestorm $ make
make -C icebox all
make[1]: Entering directory '/home/charlotte/icestorm/icebox'
python3 icebox_chipdb.py -3 > chipdb-384.new
mv chipdb-384.new chipdb-384.txt

[... lots of output ...]

cc -MD -MP -O2  -Wall -std=c99 -I/home/charlotte/.local/include    -c -o mpsse.o mpsse.c
cc -o iceprog  iceprog.o mpsse.o -lm -lstdc++ -lftdi
make[1]: Leaving directory '/home/charlotte/icestorm/iceprog'
(venv) icestorm $ make install
for dir in icebox icepack icemulti icepll icebram icetime iceprog; do \
        make -C $dir install || exit; \
done
make[1]: Entering directory '/home/charlotte/icestorm/icebox'
mkdir -p /home/charlotte/.local/share/icebox

[... lots of output ...]

mkdir -p /home/charlotte/.local/bin
cp iceprog /home/charlotte/.local/bin/iceprog
make[1]: Leaving directory '/home/charlotte/icestorm/iceprog'
(venv) icestorm $

If you have an iCEBreaker, at this stage you can try using iceprog to say hi:

(venv) icestorm $ iceprog -t
init..
cdone: high
reset..
cdone: low
flash ID: 0xEF 0x40 0x18 0x00
cdone: high
Bye.
(venv) icestorm $

Troubleshooting

The following error indicates the device wasn’t found by iceprog:

init..
Can't find iCE FTDI USB device (vendor_id 0x0403, device_id 0x6010 or 0x6014).
ABORT.

macOS users, check System Information → USB. If you don’t see the iCEBreaker listed, check your connections and consider trying a different USB cable, adaptor or hub, as appropriate.

Linux users, check lsusb. If you can see something with ID 0403:6010, that’s good. If it identifies itself as an iCEBreaker, even better. You may need a udev rule to ensure the device node is writable by your user.

Create the file /etc/udev/rules.d/53-lattice-ftdi.rules with the following content:

ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6010", MODE="0660", GROUP="plugdev", TAG+="uaccess"

This will make any device with ID 0403:6010 writable by the group plugdev. Check the output of the id command to verify your user groups:

(venv) icestorm $ id -Gn
charlotte adm sudo plugdev
(venv) icestorm $

If plugdev is listed somewhere, you’re good. Otherwise, add yourself to the group. (e.g. sudo adduser $(whoami) plugdev) After this, unplug and replug for the new rule to take effect.

WSL 2 users (or recalcitrant Windows users) should also consult the footnote⁴.

nextpnr

Ensure Project IceStorm (↴) is installed first.

Fetch nextpnr and install the appropriate § Prerequisites. Then, check out the specific instructions for nextpnr-ice40. We’ll need to adapt them slightly.

We specify the iCE40 arch, the install prefix, the install prefix for Project IceStorm, the root directory for our active Python installation, and finally, a runtime search path to add to the final binary. This is because nextpnr will link against our Python install, but our Python install’s shared libraries aren’t on the system search path.

(venv) nextpnr $ cmake . -DARCH=ice40 \
                 -DCMAKE_INSTALL_PREFIX=$HOME/.local \
                 -DICESTORM_INSTALL_PREFIX=$HOME/.local \
                 -DPython3_ROOT_DIR="$(asdf where python)" \
                 -DCMAKE_INSTALL_RPATH="$(asdf where python)"/lib
-- Building with IPO
-- Found Python3: /home/charlotte/.asdf/installs/python/3.11.4/bin/python3 (found suitable version "3.11.4", minimum required is "3.5") found components: Interpreter
-- Found Python3: /home/charlotte/.asdf/installs/python/3.11.4/include/python3.11 (found suitable version "3.11.4", minimum required is "3.5") found components: Development Development.Module Development.Embed
-- Found Boost: /usr/include (found version "1.74.0") found components: filesystem program_options iostreams system thread regex chrono date_time atomic
-- Found Boost: /usr/include (found version "1.74.0") found components: program_options filesystem system
-- Configuring architecture: ice40
-- Enabled iCE40 devices: 384;1k;5k;u4k;8k
-- Found Python3: /home/charlotte/.asdf/installs/python/3.11.4/bin/python3 (found suitable version "3.11.4", minimum required is "3.5") found components: Interpreter
-- IceStorm install prefix: /home/charlotte/.local
-- icebox data directory: /home/charlotte/.local/share/icebox
-- Using iCE40 chipdb: /home/charlotte/nextpnr/ice40/chipdb
-- Configuring architecture: ecp5
-- Enabled ECP5 devices: 25k;45k;85k
-- Trellis install prefix: /home/charlotte/.local
-- Trellis library directory: /usr/local/lib/trellis
-- Trellis data directory: /home/charlotte/.local/share/trellis
-- Using ECP5 chipdb: /home/charlotte/nextpnr/ecp5/chipdb
-- Configuring done
-- Generating done
-- Build files have been written to: /home/charlotte/nextpnr
(venv) nextpnr $ make -j8
[  2%] Generating chipdb/chipdb-384.bba
[  2%] Building CXX object bba/CMakeFiles/bbasm.dir/main.cc.o
[  4%] Generating chipdb/chipdb-1k.bba
[  5%] Linking CXX executable bbasm

[... lots of output ...]

[ 97%] Building CXX object CMakeFiles/nextpnr-ice40.dir/ice40/pack.cc.o
[ 98%] Building CXX object CMakeFiles/nextpnr-ice40.dir/ice40/pcf.cc.o
[100%] Linking CXX executable nextpnr-ice40
[100%] Built target nextpnr-ice40
(venv) nextpnr $ make install
[  7%] Built target chipdb-ice40-bbas
[ 10%] Built target bbasm
[ 17%] Built target chipdb-ice40-bins
[ 32%] Built target chipdb-ice40
[100%] Built target nextpnr-ice40
Install the project...
-- Install configuration: "Release"
-- Installing: /home/charlotte/.local/bin/nextpnr-ice40
-- Set runtime path of "/home/charlotte/.local/bin/nextpnr-ice40" to "/home/charlotte/.asdf/installs/python/3.11.4/lib"
(venv) nextpnr $

Test the installed binary to make sure it works.

(venv) nextpnr $ nextpnr-ice40
"nextpnr-ice40" -- Next Generation Place and Route (Version nextpnr-0.6-29-g54b20457)

General options:
  -h [ --help ]                         show help
  -v [ --verbose ]                      verbose output
  -q [ --quiet ]                        quiet mode, only errors and warnings
                                        displayed
  --Werror                              Turn warnings into errors
  -l [ --log ] arg                      log file, all log messages are written
                                        to this file regardless of -q

[... lots of output ...]

  --opt-timing                          run post-placement timing optimisation
                                        pass (experimental)
  --tmfuzz                              run path delay estimate fuzzer
  --pcf-allow-unconstrained             don't require PCF to constrain all IO

(venv) nextpnr $

At this point our Amaranth can now use our installed tooling to program the iCEBreaker. The board definitions we installed earlier can be executed directly to program a test blink gateware—doing this exercises the full toolchain. Verify:

(venv) nextpnr $ python -m amaranth_boards.icebreaker
init..
cdone: high
reset..
cdone: low
flash ID: 0xEF 0x40 0x18 0x00
file size: 104090
erase 64kB sector at 0x000000..
erase 64kB sector at 0x010000..
programming..
done.
reading..
VERIFY OK
cdone: high
Bye.
(venv) nextpnr $

SymbiYosys

Formal verification can be orchestrated with SymbiYosys. To get started with formal verification and Amaranth, have a look at Robert Baruch’s graded exercises for Amaranth HDL, which start with formal methods from the very first exercise. They use the tools we install here.

SymbiYosys is a relatively simple frontend, so fetch the repo and install. It also has a Python dependency, click. Check that sby -h doesn’t give an error:

(venv) sby $ make PREFIX=$HOME/.local install
mkdir -p /home/charlotte/.local/bin
mkdir -p /home/charlotte/.local/share/yosys/python3
cp sbysrc/sby_*.py /home/charlotte/.local/share/yosys/python3/
sed -e 's|##yosys-program-prefix##|"''"|' < sbysrc/sby_core.py > /home/charlotte/.local/share/yosys/python3/sby_core.py
sed 's|##yosys-sys-path##|sys.path += [os.path.dirname(__file__) + p for p in ["/share/python3", "/../share/yosys/python3"]]|;' < sbysrc/sby.py > /home/charlotte/.local/bin/sby
chmod +x /home/charlotte/.local/bin/sby
(venv) sby $ pip install click
Collecting click
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
Installing collected packages: click
Successfully installed click-8.1.3
(venv) sby $ sby -h
usage: sby [options] [<jobname>.sby [tasknames] | <dirname>]

positional arguments:
  <jobname>.sby | <dirname>

[... lots of output ...]

  --init-config-file INIT_CONFIG_FILE
                        create a default .sby config file
(venv) sby $

Check sby -h doesn’t give an error.

Z3

Z3 is a theorem prover — it does the heavy lifting of formal verification. Clone the repo; we’re going to follow the CMake instructions. The defaults are all good, except for the install prefix:

(venv) z3 $ mkdir build
(venv) z3 $ cd build
(venv) build $ cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/.local
-- The CXX compiler identification is GNU 12.2.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Z3 version 4.12.3.0

[... lots of output ...]

-- Configuring done
-- Generating done
-- Build files have been written to: /home/charlotte/z3/build
(venv) build $ make -j8
[  0%] Building CXX object src/util/CMakeFiles/util.dir/approx_set.cpp.o
[  0%] Building CXX object src/util/CMakeFiles/util.dir/approx_nat.cpp.o
[  0%] Building CXX object src/util/CMakeFiles/util.dir/debug.cpp.o

[... lots of output ...]

[ 98%] Linking CXX shared library ../libz3.so
[100%] Linking CXX executable ../../z3
[100%] Built target shell
[100%] Built target libz3
(venv) build $ make install
[  6%] Built target util
[  8%] Built target params
[  9%] Built target polynomial
[  9%] Built target automata

[... lots of output ...]

-- Installing: /home/charlotte/.local/include/z3_spacer.h
-- Installing: /home/charlotte/.local/include/z3_version.h
-- Installing: /home/charlotte/.local/bin/z3
(venv) build $

Done.

Overview

You’re now ready to write and deploy gateware, with a toolchain selected, built and installed by yourself in a self-contained and repeatable way.

There are many ways to pivot from here.

You need a different Python version.

asdf and virtual environments make this easy.

You want to understand Amaranth better.

You can modify your editable install directly, adding print debugging.

You need to write pure Verilog.

You can drive Yosys yourself.

You want to understand decisions made by Yosys better.

You can step-through debug Yosys: run your Amaranth build once, then invoke Yosys with your debugger of choice using the arguments from the generated build/build_top.sh.

You need to target a different family of boards.

nextpnr supports a range of architectures.

You want to use a different solver with SymbiYosys.

If it’s supported and on your PATH, it’ll work.

Footnotes

baby’s first gateware, in fact. ↩
- Lots of random things are a little bit broken.
- Building Yosys is certainly achievable but you simply don’t wanna.
- Everything runs slower. Everything. Git runs slower. Python runs slower. Batch scripts run slower. Yosys runs slower. iceprog communicates (much!) slower.
- Think you can fix some of this by using MSYS2 or Cygwin? Now you have two problems.
There’s more that I’ve decided was better left forgotten. ↩
tl;dr: use your Linux user home directory, not your Windows user one; if CMake takes forever during configure, check if your PATH is full of /mnt/... — if it is, it’s probably searching your Windows partition very slowly (disable interop.appendWindowsPath or modify your PATH just for configure); when it comes time to flash your board, follow the guide to Connect USB devices using usbipd-win. Don’t mind the scary warning on the guide: I didn’t have to recompile my kernel even on Windows 10. ↩
Firstly, create (or edit) the file /etc/wsl.conf and ensure you have the following stanza (or know what you’re doing already):
```
[boot]
command="service udev restart"
```
Run sudo service udev restart to get udev going immediately. You can usbipd wsl detach -i 0403:6010 and then attach again instead of physically messing around with cables.

This likely only applies to Windows Steelman Enthusiasts, but you may also need to use Zadig. Use the WinUSB driver. Check “List All Devices” in the options menu. You should see two entries that correspond to the iCEBreaker — “Interface 0” and “Interface 1” — and they might identify themselves as the iCEBreaker, or something less obvious (like “Dual RS232-HS”). Make sure you use the same driver for both. When in doubt, unplug and replug. ↩

lottia notes

Fortune habits in Nix

Step 0: don’t install fortune

Step 1: write your habits file

Step 2: write a derivation that compiles your habits file

Step 3: integrate with your shell

Step 4: prophet

Step 5: next steps

TextMode Designer

Comptime, flow, and exhaustiveness

Platform-specific hacks for high-DPI

macOS 🔗 ↩

Windows 🔗 ↩

Linux 🔗 ↩

X

Wayland

Okay, but what is the actual solution?!

wtf 🔗 ↩

Git and jujutsu: in miniature

The manoeuvre: jj 🔗 ↩

The manoeuvre: git 🔗 ↩

Remarks 🔗 ↩

Footnotes 🔗 ↩

Soft skills: jujutsu early feelings

Footnotes 🔗 ↩

Non-intrusive vtable

Aside on tags 🔗 ↩

VyxOS

Time travel, raw

Sae

TODOs 🔗 ↩

Decombing 🔗 ↩

Aside: ABTest

RV32C 🔗 ↩

Design 🔗 ↩

Time travel

Start 🔗 ↩

Change 🔗 ↩

Necromancy 🔗 ↩

Chrononautics 🔗 ↩

Amaranth to Chisel

Footnotes 🔗 ↩

Comrak on Akkoma

Future work 🔗 ↩

Reflections 🔗 ↩

Jambalam

Happy birthday!

Out-of-band experience 🔗 ↩

Modes of measurement 🔗 ↩

Footnotes 🔗 ↩

Nix revisited

Footnotes 🔗 ↩

Untangling cycles

Installing an HDL toolchain from source

Scope 🔗

Git 🔗 ↩

Python 3 🔗 ↩

venv 🔗 ↩

Amaranth 🔗 ↩

Yosys 🔗 ↩

Project IceStorm 🔗 ↩

Troubleshooting 🔗 ↩

nextpnr 🔗 ↩

SymbiYosys 🔗 ↩

Z3 🔗 ↩

Overview 🔗 ↩

You need a different Python version.

You want to understand Amaranth better.

You need to write pure Verilog.

You want to understand decisions made by Yosys better.

You need to target a different family of boards.

You want to use a different solver with SymbiYosys.

Footnotes 🔗 ↩

Step 0: don’t install `fortune`

macOS

Windows

Linux

wtf

The manoeuvre: jj

The manoeuvre: git

Remarks

Footnotes

Footnotes

Aside on tags

TODOs

Decombing

RV32C

Design

Start

Change

Necromancy

Chrononautics

Footnotes

Future work

Reflections

Out-of-band experience

Modes of measurement

Footnotes

Footnotes

Scope

Git

Python 3

venv

Amaranth

Yosys

Project IceStorm

Troubleshooting

nextpnr

SymbiYosys

Z3

Overview

Footnotes