Github activity

Here’s a simple list of reasons why you shouldn’t use someone’s github profile activity for any indication of anything whatsoever!

github squares

  1. Working at $BIGCORP - It’s common to have an enterprise install that is firewalled, airgapped or often both.
  2. Alternatives. Only recently did github offer free private repos. Bitbucket, Gitlab, etc, etc..
  3. Don’t use git - Yes, we know git is awesome but people get along perfectly fine with svn, cvs, …
  4. Decentralized development/email - Some of the most popular open source software in the world is developed over email.
  5. Only use for personal projects - some people use github only for personal projects they poke at rarely and thats OK.
  6. Only use for work - conversly some may only use for work tasks, or only some repos, etc etc. Some of the highest energy development efforts do not end up in repos, like planning and meetings.
  7. Other things I’m forgetting
  8. Bonus! My profile shows different green squares when logged in versus private tab. I have show private contributions on ..

Just because someone doesn’t have a wall of green tiles cranking out javascript everyday for a bunch of hip organizations doesn’t mean they’re not a super active developer somewhere else.

This is ranty but I sometime see connotations from github activity (particulary some low bar recruiters) and it is really disappointing to treat a single web gadget as some sort of pissmeter.

April 8, 2019

Comparison of BSD source trees

A while back I felt the need to get involved in something ‘hard’ and new for me so I delved head first into NetBSD. I covered why in detail in the ‘mental waterboarding’ post, but as a knee jerk reaction, picked NetBSD for portability and because I thought it was smallest of the big three.

I excitedly posted on the mailing list and unsurprisingly got zero response. Naturally, as soon as I publicly announced this I got busy with IRL stuff, job changes, etc etc and didn’t ever do much.

Fast forward to today and my situation is improved again and I’ve got the urge to get back to it and decided to actually do some homework and do some ballpark comparisons of the BSDs!

I compared the following source trees, excluding ports/pkgs. This will be a comparison of kernel and base userspaces provided for:

After cloning each from the github mirrors because installing CVS in 2019 is stupid, I started with raw size on disk:

➜  bsd_code_compares du -h -d1 .     
2.7G ./freebsd_src
5.7G ./netbsd_src
2.0G ./openbsd_src
 10G .

NetBSD is actually the biggest on disk. I would have expected FreeBSD.

Now for code comparisons. I used cloc in the root of the directory, with no filetype filter, so there is a bit of false positives, so take as a rough idea

OS files blank comment code
OpenBSD 53591 2622788 3365778 14912018
NetBSD 123385 6329344 8430136 34842316
FreeBSD 52867 2850675 4186693 17083775

A good follow up would be comparing just the kernel trees, comparing # of drivers, etc..

April 1, 2019

Diskpart/etcher errors - Wiping a removable/SDcard on win10

Tools like etcher use diskpart 'clean' command to nuke the partitions. On windows 10 this often will fail everytime when you try to image a disk that was previously flashed with a bigger image, it seems to be 4GB+. Etcher will just give a generic error (and its written in javascript, what could go wrong!), but diskpart is the root cause.

etcher error

If you're using diskpart by hand and run clean it will say access is denied and to check the event log or such:

DISKPART> clean
DiskPart has encountered an error: The request could not be performed because of
an I/O device error.
See the System Event Log for more information.

The event log will have an entry with something like this, which is not helpful

Cannot zero sectors on disk \\?\PhysicalDrive2. Error code: 5@0101000F

The trick is to zero the partition table so it stops getting hung up for who knows why. Of course, if you have a linux machine handy you can just write zeros to the whole device with dd, but you wouldn't be trying to use diskpart in that case anyway. (dd if=/dev/zero of=/dev/sd<5> bs=8M count=256)

So if you dont want to juggle machines or only have windows, there is a nice little freeware tool that will perform exactly the same thing, designed more for secure wiping disks by repeated writes, but you can do one pass and cancel it like 5% the way through, and then etcher will flash successfully.

The tool is "RoadKil's disk wipe" available here

August 25, 2018

Getting wine 3.0 working on ubuntu 18.04

Wine stable on ubuntu 18.04 seems to have broken auto-fetching gecko, which can break many installers with EULAs.

You can see this happen with wine iexplore https://google.com or such, you will get an error like

Could not load wine-gecko. HTML rendering will be disabled.
0009:err:mshtml:create_document_object Failed to init Gecko, returning CLASS_E_CLASSNOTAVAILABLE

I ran into it with diabloII installers not being able to show and accept EULAs.

The site has instructions explaining that the auto-installer would just grab the msis and plop em in the correct spot, which can also be done manually. Here's a quick and dirty on how to install wine and get gecko working:

  1. Install wine if you havent with sudo apt install wine-stable
  2. sudo mdkir /usr/share/wine/gecko && cd /usr/share/wine/gecko
  3. sudo wget http://dl.winehq.org/wine/wine-gecko/2.47/wine_gecko-2.47-x86_64.msi
  4. sudo wget http://dl.winehq.org/wine/wine-gecko/2.47/wine_gecko-2.47-x86.msi
  5. verify it works with wine iexplore https://google.com
July 21, 2018

Detecting simple metrics in Angular

Here is how to cover collecting metrics (or events if you prefer) such as click or touch events on DOM elements across an entire Angular2+ project.

This has the following caveats: It's designed for simple 'did the user interact' with elements type of logging, not advanced mouse trail logging and ux type of metrics. I recommend a full blown metrics product for anything that gets any more involved. Additionally, it rips through the DOM so this is not recommended on any performance sensitive elements or extremely large DOM trees.

Lastly - if you only care about input resulting in a useful action, simply log redux actions instead.

With that said here's what is needed!

  1. Rig up @HostListener for each event type we care about
  2. Grab DOM metadata programmically
  3. Attach metadata to elements we want to track
  4. Send off metadata and event info

Rigging up @HostListeners

This is probably the easiest step, and this is a really useful part of Angular that makes its very easy to capture events SPA-wide.

Lets setup listening to every click event that happens in our <body> by putting this @HostListener in our toplevel 'app' component (or similar):

NOTE: In my production code i pulled event type into an enum and recommend strongly typing all known event types you care about, but we will keep it simple here as a string 'click'.

@HostListener('body:click', ['$event'])
onBodyElementClicked(ev: Event) {
  this.reportEventMetric(ev.target, 'click');
}

We will create reportEventMetric later on. This will fire when anything is clicked anywhere in the body, so this is the only event handler we need!

Grabbing metadata programmically

Next we need to create a function that will look at whatever was clicked and determine if it's been tagged to keep track of.

For this we need to determine a metric attribute. In this example I'll use a data-metric attribute.

Now all we need to do is look for data-metric on the DOM element, BUT not just it, we want to walk up the visual tree incase the click was reported on a child element that we care less about.

private reportEventMetric(start: EventTarget, evType: string) {
    // this helper will call a recursive function to rip through DOM
    const metric = this.findDataMetricTag(start, evType);

    if (metric) {
        // once you have metrics, you can POST to server, export to csv,
        // stick in state, whatever. in this case lets just log em
        console.log(`${evType} metric: ${metric}`);
    }
}

// here is the meat recursive function that will rip through dom elements looking
// for our data attribute
private findDataMetricTag(start: EventTarget, eventType: string): string | null {
    if (!start || !(start instanceof HTMLElement)) {
        return null;
    }

    // in production code, i recommend pulling 'data-metric' to a const or readonly class prop
    const metricAttr = start.getAttribute('data-metric');
    return metricAttr || this.findDataMetricTag(start.parentElement, eventType);
}

In my first attempt, findDataMetricTag was actually 'Tags' because it would always walk up the parents and report any tags found, but this was not actually useful and more performance intensive.

Now we have a way to dig out the metric tag!

Tagging elements for metrics

This is the easiest, but most grueling last part, we will do this in the straightforward way of sticking directly in markup (better ways probably exist). Simply add the data attribute to an element for it to be reported:

<div data-metric="wizzbang-container-bot">
<!-- stuff -->
</div>

<button data-metric="do-thing-button">
</button

Once this is done, if we click on an element we should see the tag reported in console, thats it!

Bonus route information!

It's often very useful to know which route you are currently on so you don't have to bake it into every single data-metric tag. This is easy to do with Angular's Location service. Inject it and get the current route like so:

import {Location} from '@angular/common';

// .. / .. in class constructor that has `reportEventMetric:
constructor(private _location: Location) {
}

// now modify reportEventMetric to print current location like so:

private reportEventMetric(start: EventTarget, evType: string) {
    // this helper will call a recursive function to rip through DOM
    const metric = this.findDataMetricTag(start, evType);

    // this will pull out the path of current route, eg '/user' from route
    const location = this._location.path();

    // once you have metrics, you can POST to server, export to csv,
    // stick in state, whatever. in this case lets just log em
    console.log(`route '${location}' ${evType} metric '${metric}'`);
}

And there you have it, simple metrics ready to stuff in a backing store.

May 25, 2018

the curious case of SIGPIPE

Shell interaction, signals, TTYs, jobs and the like are some of the core foundations that have been around since the birth of time sharing operating systems but much of it is left in the literal 'black box', even by developers! It can be quite complex when you start from the bottom up, but lets take a look at SIGPIPE, it allows for some interesting shell interaction that often goes unnoticed but is immediately useful.

Generators

We start this example with a source of otherwise 'endless' input for all intents and purposes. So called generators are a way to generate an infinite list of values, not unlike eg, [1..] in Haskell generators

The idea is to have something that will spew values as long as something is interested. yes is probably the most simple of such programs, designed to generate 'y' that can be piped to prompts and the such. The 'spoiler alert' here is that yes somehow knows to end its input despite consisting of an infinite loop in code.

interesting 'yes' complexity side note

yes may sound like the most simple C program imaginable. It mostly is, the guts of the OpenBSD version essentially consists of this:

for (;;)
    puts("y");

(full sources: https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c)

However in the GNU version (in coreutils) is more optimized for performance and locale handling, and ends up surprisingly different, with local buffering ahead of time and such: https://github.com/coreutils/coreutils/blob/master/src/yes.c

This is worth pointing out because when we dig in later with strace, there will be 'noise' of these other operations in our investigation.

The writing mechanic, eg, puts(). writes to STDOUT which in this case has been rigged up through our job pipe. Speaking of job pipes!

Back to the pipe pipeline

So with generators and yes in mind, we can use in shell pipes, which falls into job control. TTY handling, shell interaction and job control is a very detailed beast I will not go into detail on here, but in layman's terms piping several commands together sticks em together in a job (which is within a session). This allows the whole operation to handled discreetly with signals, or 'bg', 'fg' and 'jobs' commands on any *NIX.

shell jobs

This is important to note because a job is created, a pipe handle is created to direct STDOUT (by default, you can specify which with 2>&1 and so forth) to STDIN between all the subprocesses of the job and we will see the handle id later. For our purposes, just know the job creates a pipe to funnel STDOUT through. This is not technically completely correct but if you know better put on your blinders (especially before seeing these doodles) shell pipe human centipede

(sorry, done playing with wonky watcom since i can't analog-type anymore)

show me the yes

Running yes in a basic pipe, something like this:

dmh@beer-disposal:~$ yes | head
y
y
y
y
y
y
y
y
y
y

if only the people behind asking for raises during performance reviews were more like yes

Prints exactly 10 'y'. Why 10? thats the default number of lines head will read as you can see in man head Yet, yes somehow knows to then terminate since no one is interested in reading the output anymore.

It seems to magically know when head is done. Changing the amount of lines head hovers up with eg, yes | head -n2 works as expected, printing two 'y' lines and going back to your shell prompt normally.

But yes spews 'y' endlessly! How does it know when to kindly STFU? This is handled with SIGPIPE! We can see this in the signal.h docs

"The SIGPIPE section denotes default action is 'T' (terminate) and 'Write on a pipe with no one to read it.'"

the write syscall docs also specify in failure conditions:

"An attempt is made to write to a pipe or FIFO that is not open for reading by any process, or that only has one end open. A SIGPIPE signal shall also be sent to the thread"

wait a minute, i thought yes used puts

Before we get lost in the weeds, things like printf and puts use the write() syscall to actually.. write output. This is handled by the C runtime library!

breakdown of what happens

Dry docs aside and ignoring non-essential details like buffered read/writes, here is what happens:

  1. yes writes batches of 'y' repeatedly to STDOUT (which behind the scenes is a job pipe)
  2. head reads from STDIN (the pipe until it satisfies 10 lines and then exits (closing its handle to the pipe)!
  3. yes is still furiously trying to write to same pipe, next write since head exits returns -1.
  4. yes receives SIGPIPE signal itself and also exits

This ignores some less important details like STDOUT line buffering and buffered read/write causing a difference in read/written bytes but it is not relevant to get the point of what is happening.

Digging deeper

We can see this by running the pipe operation through strace:

dmh@beer-disposal:~$ strace -f sh -c 'yes | head'

Here is the important parts highlighted, pid 6541* is yes and pid 6542** is head

strace

On a side note, you may notice lots of other things going on in the strace, that is because this is GNU coreutils yes, which as noted above does a lot of other 'stuff' in name of locale handling and performance.

A drastically better looking strace can be achieved on BSD systems for the motivated

Ok whats your point

The next time a haskell hipster is spouting the benefits of lazily computed infinite lists, let them know unix has had a pragmatic version of the same thing since before they were probably born!

May 23, 2018

Windows 10 Wifi tweaks

Wifi has never been great on windows 10, especially after the latest 'fall creator update'. One of my wifi cards no longer works at all, despite working on this same exact version of windows before a clean install!

Anyway, I've found a few ways that seem to make the wifi slightly more consistent, so here they are

Step zero: Wifi adapter name

We will need the short name of the wifi adapter and the easiest way is with **netsh*.

To list wlan settings run the following in a cmd or powershell (Win+X):

netsh wlan show settings

PS C:\Users\Dexter> netsh wlan show settings

Wireless LAN settings
---------------------
    Show blocked networks in visible network list: No

    Only use GP profiles on GP-configured networks: No

    Hosted network mode allowed in WLAN service: Yes

    Allow shared user credentials for network authentication: Yes

    Block period: Not Configured.

    Auto configuration logic is disabled on interface "Wi-Fi 3"
    MAC randomization not available on interface Wi-Fi 3

Look for the quoted part after interface, in this case my adapter name is Wi-Fi 3. Alternatively, you can go to

Control Panel -> Network Connections

And that will have the same name. We'll need it for the rest of the steps.

First: Wifi autoconfig on/off bats

Save these two in batch files so you can easily run them as needed:

WIFI_AUTO_ON.bat netsh wlan set autoconfig enabled=yes interface="Wi-Fi 3"

WIFI_AUTO_OFF.bat netsh wlan set autoconfig enabled=no interface="Wi-Fi 3"

Replace "Wi-Fi 3" with the adapter name from the first step!

Now you can right click on WIFI_AUTO_OFF.BAT and run as admin before playing an online game. For me, this seems to help with the persistent, roughly 1 minute apart ping spikes.

Second: disable tuning heuristics

This one is a bit harder to quantify but in my highly non-scientific testing, my ping appeared to be more stable overall after turning off auto tuning. Note this is machine wide and seems to be related to TCP window sizes. I've encounted no side effects but..

netsh int tcp set heuristics disabled
netsh int tcp set global autotuninglevel=disabled
netsh int tcp set global rss=enabled

Disable dat heuristic tuning

Hopefully this helps others eek out a remotely usable wifi experience on windows.

April 12, 2018

NetBSD cross-compiling tutorial

Lets build NetBSD starting from nothing!

Thanks the to the great build.sh system I've already gushed about, cross-compiling NetBSD from any other POSIX'y host is cake.

However, the docs are slightly out of date, and everything needed is scattered across a few pages which may not be obvious for new-comers. For example, grabbing 'sets' which is parlance for gzips of source code, is on a different page, and the kernel compile page is missing some new flags.

Similarly, man pages are great if you already have an idea of what you're looking for. But I'll save that diatribe for another day.

There is not much absolute-beginner orientated task-orientated documentation, too. Certainly the BSDs are niche and demand expertise but there is no reason to unnecessarily burden the learning curve for passerby's. This increases adoption!

I've also learned a few other things since writing my previous compiling notes. Particularly, the drivers in use when NetBSD is a virtualbox guest cause noticeably slower disk performance at the moment. I highly recommend to build sources from anything but a VM

So with that said, this will be a tutorial for building NetBSD from a GNU/Linux host. The instructions should be very similar for Mac and other BSDs.

Also, this will cover the 7.1.2 release, not current (WIP) sources To recap:

Step 1: grab source sets

Navigate your big nasty web browser to the source sets directory here:

https://ftp.netbsd.org/pub/NetBSD/NetBSD-7.1.2/source/sets/

Grab all the tgz, you can skip xsrc.tgz unless you want to compile X windows.

I recommend saving all these in a tidy folder like ~/netbsd_build/7.1.2

Next, untar em all, eg manually:

$ pwd
~/netbsd_build/7.1.2
$ tar xvzf gnusrc.tgz
$ tar xvzf sharesrc.tgz
$ tar xvzf src.tgz
$ tar xvzf syssrc.tgz
$ tar xvzf xsrc.tgz  #(again, only if you want to compile X)

if this pissed you off: write a shell for with wildcard expansion to do it in a oneliner!

Once that's all done you should end up with something like this:

dexter@slag:~/netbsd_build/7.1.2$ tree -L 2
.
├── gnusrc.tgz
├── sharesrc.tgz
├── src.tgz
├── syssrc.tgz
└── usr
    └── src

2 directories, 4 files
dexter@slag:~/netbsd_build/7.1.2$

the funky sub usr/src dir is ok. Navigate into that directory for the rest of the instructions $ cd usr/src

Next, we need to make a directory to output objects during build, if we dont do this we will get errors. It doesnt really matter where, just make the directory in the usr/src subdir we are already in:

$ mkdir obj

We're ready to start compiling stuff!

Building toolchain

The first step to build NetBSD is to build the compiler it expects to use. Long story short, dont expect, or try to use your host compiler. It's much easier (albeit it can be slow on older machines) to use the provided gcc sources.

To do so, fire off build.sh like so:

$ ./build.sh -U -m amd64 -O obj -j17 tools

A quick explaination of the flags:

It will grind a while, especially if you're on older hardware but you should eventually get an output like this

===> Tools built to /home/dexter/netbsd_build/7.1.2/usr/src/obj/tooldir.Linux-4.13.0-38-generic-x86_64
===> build.sh ended:      Sun Apr  8 00:15:25 MDT 2018
===> Summary of results:
     build.sh command:    ./build.sh -U -m amd64 -O obj -j17 tools
     build.sh started:    Sun Apr  8 00:12:01 MDT 2018
     NetBSD version:      7.1.2
     MACHINE:             amd64
     MACHINE_ARCH:        x86_64
     Build platform:      Linux 4.13.0-38-generic x86_64
     HOST_SH:             /bin/sh
     MAKECONF file:       /etc/mk.conf (File not found)
     TOOLDIR path:        /home/dexter/netbsd_build/7.1.2/usr/src/obj/tooldir.Linux-4.13.0-38-generic-x86_64
     DESTDIR path:        /home/dexter/netbsd_build/7.1.2/usr/src/obj/destdir.amd64
     RELEASEDIR path:     /home/dexter/netbsd_build/7.1.2/usr/src/obj/releasedir
     Updated makewrapper: /home/dexter/netbsd_build/7.1.2/usr/src/obj/tooldir.Linux-4.13.0-38-generic-x86_64/bin/nbmake-amd64
     Tools built to /home/dexter/netbsd_build/7.1.2/usr/src/obj/tooldir.Linux-4.13.0-38-generic-x86_64
     build.sh ended:      Sun Apr  8 00:15:25 MDT 2018
===> .

Now we're ready to move on

Building a kernel redux

To kick out the kernel, we follow suite with a slightly different command:

$ ./build.sh -U -m amd64 -O obj -j17 kernel=GENERIC

The new option kernel=GENERIC denotes which amd64 kernel config file we want to build. These configs live in the subdir sys/arch/<arch>/conf/ so in our case sys/arch/amd64/conf/GENERIC is the file. Ideally, you'd copy and modify this file to your desire. But for this example lets compile the stock GENERIC. While its out of scope for this scenic tour, pique your interest with the 'makeoptions COPTS` line.

Same looking output:

--- netbsd ---
#      link  GENERIC/netbsd
/home/dexter/netbsd_build/7.1.2/usr/src/obj/tooldir.Linux-4.13.0-38-generic-x86_64/bin/x86_64--netbsd-ld -Map netbsd.map --cref -T /home/dexter/netbsd_build/7.1.2/usr/src/sys/arch/amd64/conf/kern.ldscript -Ttext 0xffffffff80100000 -e start -z max-page-size=0x100000 -X -o netbsd ${SYSTEM_OBJ} ${EXTRA_OBJ} vers.o
NetBSD 7.1.2 (GENERIC) #1: Sun Apr  8 00:33:55 MDT 2018
   text    data     bss     dec     hex filename
14227197     654556  593920 15475673     ec23d9 netbsd
===> Kernels built from GENERIC:
  /home/dexter/netbsd_build/7.1.2/usr/src/obj/sys/arch/amd64/compile/GENERIC/netbsd
===> build.sh ended:      Sun Apr  8 00:33:56 MDT 2018
===> Summary of results:
     build.sh command:    ./build.sh -U -m amd64 -O obj -j17 kernel=GENERIC
     build.sh started:    Sun Apr  8 00:33:12 MDT 2018
     NetBSD version:      7.1.2
     MACHINE:             amd64
     MACHINE_ARCH:        x86_64
     Build platform:      Linux 4.13.0-38-generic x86_64
     HOST_SH:             /bin/sh
     MAKECONF file:       /etc/mk.conf (File not found)
     TOOLDIR path:        /home/dexter/netbsd_build/7.1.2/usr/src/obj/tooldir.Linux-4.13.0-38-generic-x86_64
     DESTDIR path:        /home/dexter/netbsd_build/7.1.2/usr/src/obj/destdir.amd64
     RELEASEDIR path:     /home/dexter/netbsd_build/7.1.2/usr/src/obj/releasedir
     Updated makewrapper: /home/dexter/netbsd_build/7.1.2/usr/src/obj/tooldir.Linux-4.13.0-38-generic-x86_64/bin/nbmake-amd64
     Building kernel without building new tools
     Building kernel:     GENERIC
     Build directory:     /home/dexter/netbsd_build/7.1.2/usr/src/obj/sys/arch/amd64/compile/GENERIC
     Kernels built from GENERIC:
      /home/dexter/netbsd_build/7.1.2/usr/src/obj/sys/arch/amd64/compile/GENERIC/netbsd
     build.sh ended:      Sun Apr  8 00:33:56 MDT 2018
==> .

The actual netbsd kernel binary will be sitting in our obj/sys/arch/amd64/compile/GENERIC/ dir. To actually install it you'd just copy it over your existing netbsd kernel.

dexter@slag:~/netbsd_build/7.1.2/usr/src$ file obj/sys/arch/amd64/compile/GENERIC/netbsd
obj/sys/arch/amd64/compile/GENERIC/netbsd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, for NetBSD 7.1.2, not stripped

Userland

Next is userland. no frills here, you will see the pattern

$ ./build.sh -U -m amd64 -O obj -j17 release

The only new option is just release. If you want to build X also pass in -x

Further reading

At this point I've just overlapped the official guide on building, which has all the details. Check it out once you are ready to make modifications and actually install your new bins!

https://www.netbsd.org/docs/guide/en/chap-build.html

April 6, 2018

Reading linux mouse input with golang

I think mouse input in linux is generally awful. It always feels terrible in any window manager and I have begun digging into the whys. It seems mostly related with X adding acceleration. In my ventures I learned how to read from a mouse device without X. Lets read the mouse input from linux!

In UNIX'y operating systems everything is a file which many people seem to enjoy shouting off at irrelevant times with no context. For our purposes, this means the mouse will be represented as a file somewhere.

If you only have one mouse (why would you have two? freak) you can check /dev/input/mouse0

$ file /dev/input/mouse0

/dev/input/mouse0: character special (13/32)

With that we can see the mouse is a so called 'special' which is short for a 'special file', aka device file

A quick and dirty example in go to get the relative x/y update and whether or not left/right/middle is down would look something like this. Note this is blocking and could be drastically better

package main

import (
    "os"
    "log"
    "fmt"
)

// TODO https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/input.h#n28

func main() {
    f, err := os.OpenFile("/dev/input/mouse0", os.O_RDONLY, 0600)
    if err != nil {
        log.Fatal(err)
    }

    data := make([]byte,  3)

    for {
        nr, err := f.Read(data)
        if err != nil {
            log.Fatal(err)
        }

        if nr == 0 {
            break
        }
        
        left := data[0] &amp; 0x1 != 0;
        right := data[0] &amp; 0x2 != 0;
        middle := data[0] &amp; 0x4 != 0;
        x := int8(data[1])
        y := int8(data[2])
        fmt.Printf("\r x=%d y=%d left=%v right=%v middle=%v ",
            x, y, left, right, middle)
    }
}

Assuming you have your GOPATH setup correctly (or use the default of ~/go), save this in a package called eg, linux-mouse-events.

Then you can test it like so (CTRL-C to exit):

$ go install linux-mouse-events
$ sudo ~/go/bin/linux-mouse-events
x=1 y=0 left=false right=false middle=false ^C

Amazing! Next we will move to timing evdev events to see if anything mucks with raw mouse report rate

April 3, 2018

'Le Potato' boot serial port

The AML-S905X-CC (Le Potato) is a strange beast. It doesn't have reams of friendly documentations and is fairly barebones.

I decided to start looking into where u-boot lives, where early boot output goes and things of that nature.

So off I went and plugged in a USB serial cable to the usual spot for a raspberry pi style header. From top right, its VCC VCC Ground UART_TX UART_RX.

attempt 1

Cracked open putty, serial 115200 baud lets go. Not my first rodeo..

doh

Never said I was able to ride the bull 8 seconds. Wrong baud rate? That would be a first for me. Maybe it has different headers. I go to check pin-out, but they do not provide a pinout!

They do however provide a schematic. I've mirrored it HERE because they served it from drive so who knows how long that will be around.

Huh, it mostly matches the raspberry pi layout..

header schematic

I decided to quickly flip around to the CPU(s) section of the schematic to make sure UART_A_TX and UART_A_RX are the right thing. In the process I discovered they have broken out the IO pins for UART in a separate spot named 'UART DEBUG' (promising!) and they are labelled differently as 'linux rx/tx':

uart debug

physical location next to hdmi

Instead of wasting time guessing UART baud rate incase its something weird I decided to crack out a saleae logic analyzer. By the way, I cannot recommend these things enough, they are amazing.

I plugin the UART pins and add async serial analyzers with 'autobaud' on, so they will scan smallest pulses as they are received and adjust baud.

hooked

serial analyzer

This works without issue and the reported baud setting is ~116279 after a capture, so 115200 is probably correct.

Plugging the usual USB serial cable into the debug UART at 115200 shows us glorious U-boot output and works.

serial analyzer

This is a custom u-boot that I am unsure is completely upstreamed. I've never seen so much stuff printed out before uboot header.

Takeaways

As expected, the le potato will require digging. aside from one (active) forum, you're basically on you're own. On the plus side the developers seem engaged with upstream, have their sources on github and provided schematics.

March 26, 2018

NetBSD features you may have missed

It's no secret NetBSD is the smallest of the three contemporary modern BSDs (FreeBSD, OpenBSD, NetBSD, intentionally ignoring PC-BSD/TrueOS). Many don't give NetBSD any more thought than 'oh its the one that can run on a toaster' (which is true as long as it has a MMU :)

the netbsd poster-child toaster-child (Full article here )

There are some interesting features that are useful on desktops as well as embedded that deserve being mentioned. I list some of the features that caught my eye in no particular order. Also in no way are these unique to NetBSD (well mostly), but some may surprise you.

Lua kernel / ring0 code

I'm not sure how popular it is or if its really used, but it is a possibility to write kernel modules in Lua. This is pretty darn cool and lua support in the kernel is the fruits of a previous Google Summer of Code (GSoC) project.

For Lua to work, you need to load the lua and lua sytem modules:

# modload lua
# modload luasystm

Then you can create a Lua state and load a script like so:

# luactl create state1
# luactl require state1 systm
# luactl load state1 ./luahello.lua

You can find full examples here Def checkout intro(9lua) too.

CGD disk/partition encryption

CGD (Cryptographic device driver) lets you encrypt disks at an entire disk or partition level. Of course, this is nothing new, but it has more flexibility for key sizes and available ciphers versus, eg, OpenBSD. This may no longer be true

rump kernel drivers in userspace

juicy rump

NetBSD Rump kernels offer many benefits, of particular interest is being able to compile kernel drivers and test it in seconds without using a VM. In simplified terms, it is basically the various drivers that usually live in kernel space, but running as separate entities in the rump kernel userspace.

Rump kernels are not an operating system specific concept, for background see What is a rump kernel? NetBSD 'anykernel' is (was?) the first implementation of a usable rump kernel though. It has a funny name to boot

pkgsrc can do some cool stuff

I have to admit I'm just starting to dip my toes in pkgsrc. It's a solid package manager, downloading and compiling sources for you and managing deps, as you would expect of a package manager in 2018. It has some lesser known auxiliary features:

Veriexec - Ring0 binary/file/data verification

In layman terms, veriexec compares binaries and files to a pre-configured whitelist of hashes and refuses to run any that diverge. While this would require a bit of setup (and would be annoying for casual daily desktop use), you could really lock down a machine with this setup that would be resilient to even root privilege escalations.

With the state of MD5 and SHA1 hashes being worthless for integrity and having vulnerabilities and proof of collision respectively, it supports SHA256, SHA384, SHA512, and RMD160 hashing. There are a few levels of strictness which can make the machine dynamically learn or report in an IDS style of operation as well.

Sane config/directory hierarchies and other non-technical things

This is not so much a feature as my personal preferences, but on NetBSD you get things like a sane, centralized rc.d startup system. No 6 runlevels, no systemd, no having to rifle around to find defaults. /etc/rc.d after looking at /etc/defaults/rc.conf and be done with it.

Additionally, third party stuff always ends up in /usr/pkg and their configs go in /usr/pkg/etc. While GNU/Linux isnt awful, I am annoyed switching distros and having no idea where things go, install to, overwrite, etc.

Being able to install an entire system (with X) using an installer under 350mb is unheard of in this day and age too. I'm pretty sure the Nvidia drivers are bigger than that now.

March 25, 2018

NetBSD docs tarpits and kernel compiling

While my VM setup is the way to go I eventually got NetBSD-current going on the spare machines I previously mentioned. It would only be fair to mention that when NetBSD 8 comes out, it will work much better on amd64 systems.

With that said, I've been trucking along to basic usage and compiling the system. For package management, I've been just grabbing binary packages as needed with pkgin, not really interested in trying to compile big stuff with pkgsource yet.

As a beginner, I found the package management scenario presented in a burdened way. The flexibility of (having to) picking your package trees, binary versus sources, pkgsource config, pulling in CVS etc increases the cognitive load. I just needed tmux! So I only do basic pkgin usage for now.

Scan mailing lists

It's worth mentioning, even if you have no intention of running NetBSD-current or getting into the code, if you run into issues, visually scan or search the mailing list archives, someone else likely has already griped about your issue if its even remotely common.

There are actually two approaches to this, you can search GNATS, but its kinda slow to query, and you kind of have to know what you're looking for in issues. Or just check netbsd-bugs and netbsd-users mailing lists for a decent litmus-test.

netbsd-bugs

NetBSD-bugs will be slightly exaggerated noise with GNATS auto replies, but if you're having an issue it will likely be beaten to death by faithful testers here.

Moving on to grabbing tarballs of sources (sets as it were) and compiling the kernel, I googled and ended up at the NetBSD wikis for various things, eg:

I plodded through and things immediately went very badly and in hindsight I should have checked sooner..

oops

long story short: DO NOT USE THE WIKI (for stuff in the guide). It's basically redundant compared to the guide, and has largely fallen to the wayside. I think the guide is in tree next to sources and was mostly up to date.

I still ran into minor issues with the current release and reported them. But overall there was no show stoppers for the kernel compiling section of the guide.

System and kernel compiling

One minor issue I had was the build.sh instructions not working as prescribed. But looking at the error output, it was easy to figure out. For people less weathered it might not have been trivial to know to create an 'obj' folder or specify it and run as -U for non-root users. But then again NetBSD demands a bit of knowledge up front. For example (this is after you have dong config <cfgfile>, GENERIC on amd64 here), here's how I got going. Change -j9 to number of cores in your machine +/- 1 for best results. There is a useful file, BUILDING sitting in /usr/src once you have unzipped sets that has full explanations and useful examples. Alternatively the 'building the kernel manually' guide works fine still. I'm interested in using the excellent cross compiling capabilities of build.sh though.

$ cd /usr/src
$ mkdir obj
$ ./build.sh -U -u -O obj -j9 tools

(this will take a while, gcc is a bear of a compile)

$ ./build.sh -U -u -O obj -j9 kernel=GENERIC

--- netbsd ---
#      link  GENERIC/netbsd
/usr/src/obj/tooldir.NetBSD-7.1.2-amd64/bin/x86_64--netbsd-ld -Map netbsd.map --cref -T /usr/src/sys/arch/amd64/conf/kern.ldscript -Ttext 0xffffffff80100000 -e start -z max-page-size=0x100000 -X -o netbsd ${SYSTEM_OBJ} ${EXTRA_OBJ} vers.o
NetBSD 7.1.2 (GENERIC) #1: Fri Mar 23 13:51:54 MDT 2018
   text    data     bss     dec     hex filename
14223597         654556  593920 15472073         ec15c9 netbsd
===> Kernels built from GENERIC:
  /usr/src/obj/sys/arch/amd64/compile/GENERIC/netbsd
===> build.sh ended:      Fri Mar 23 13:51:57 MDT 2018
===> Summary of results:
         build.sh command:    ./build.sh -U -O obj -j9 kernel=GENERIC
         build.sh started:    Fri Mar 23 13:49:31 MDT 2018
         NetBSD version:      7.1.2
         MACHINE:             amd64
         MACHINE_ARCH:        x86_64
         Build platform:      NetBSD 7.1.2 amd64
         HOST_SH:             /bin/sh
         MAKECONF file:       /etc/mk.conf (File not found)
         TOOLDIR path:        /usr/src/obj/tooldir.NetBSD-7.1.2-amd64
         DESTDIR path:        /usr/src/obj/destdir.amd64
         RELEASEDIR path:     /usr/src/obj/releasedir
         Updated makewrapper: /usr/src/obj/tooldir.NetBSD-7.1.2-amd64/bin/nbmake-amd64
         Building kernel without building new tools
         Building kernel:     GENERIC
         Build directory:     /usr/src/obj/sys/arch/amd64/compile/GENERIC
         Kernels built from GENERIC:
          /usr/src/obj/sys/arch/amd64/compile/GENERIC/netbsd
         build.sh ended:      Fri Mar 23 13:51:57 MDT 2018
===> .

Finally, I can start digging in. This compile time certainly helps! Modern computing rules! This VM is not even on a SSD. I'm hopeful if I moved it and gave it a few more cores it could build in 60 seconds. A full release build of tools, userland and X across 16 cores probably still takes 10s of minutes though.

March 22, 2018

Installing NetBSD: amd64 pain ahoy

As covered in a previous post, I decided to deep dive a more UNIX'y operating system, ultimately NetBSD. And so it begins

When I want to learn something new, I go all-in. Diving into NetBSD, I wanted to go a 'total immersion' route, mostly because I'm fortunate enough to have lots of spare machines around I could test on and thought it would be best to use on real hardware. And focus on that machine with no distractions, just man pages like times past.

I also went all-in on coke cherry zero at my battlestation as you will see. This stuff is like nectar from the sugar free gods.

slurp

I picked NetBSD because I wanted a challenge and I wasn't exactly expecting smooth sailing but so far I've got challenges in all the places I was not expecting!

Sometimes I get irrationally mad or frustrated with things when in a hurry to get started, so this is more me aligning reality with expectations and documenting my progress.

Let me say this is not an attack on NetBSD, I understand that it's a smaller project compared to FreeBSD and OpenBSD and not even to mention GNU/Linux. It's not like companies are paying people to hack out drives for NetBSD.

Hardware

First, here's some of the hardware I've seriously tried (and remembered to write stuff down on) so far:

I decided to start with the latest regular, non-development release NetBSD 7.1.2, the newest release (5 days old at time of writing)

Getting diagnostics during installation

But first an aside: know how to dig info off a machine during installation!

This may not be obvious for newer experimenters like myself, but it didn't occur to me how difficult it would be to grab outputs of dmesg off a machine during installation to share for help.

The easiest way is to have an USB stick formatted as FAT and insert it after dropping to shell from installer menu (if you can). I was booting off the installation USB stick, in that case I needed two. Do not have both plugged in when you boot, just installation media.

You can boot the install media and immediately bail to shell once you see the menu. Once there you can mount the USB stick like so

# mkdir /mnt/usbstick2

(plugin usb stick, at this point you might see in console, or:)

# dmesg

... check the latest message about mounting umass on device /dev/sd1 or such. Mine looked like this

umass1 at uhub2 port 7 configuration 1 interface 0
umass1: USB2.0 Flash Disk, rev 2.00/0.00, addr 5
umass1: using SCSI over Bulk-Only
scsibus1 at umass1: 2 targets, 1 lun per target
sd1 at scsibus1 target 0 lun 0: <USB2.0, Flash Disk, 2.60> disk removable
sd1: fabricating a geometry
sd1: 961 MB, 961 cyl, 64 head, 32 sec, 512 bytes/sect x 1968128 sectors
sd1: fabricating a geometry
sd1: mbr partition exceeds disk size

Now we know the device is 'sd1', so the first partition would be `sd1a':

# mount_msdos /dev/sd1a /mnt/usbstick2
# dmesg > /mnt/usbstick2/dmesg.txt
# sync (or you can do umount /dev/sd1a)

If you're coming from a gnu/linux background like me, take heed and do not typo it sda1 a million times, because the backspace key will NOT work correctly and you'll go mad.

Importantly, dont forget the sync or umount command. If you rip out the USB stick before hand, the file could be not there, or likely, present but empty.

Installation - laptop

Starting with the laptop, this one was very frustrating to figure out at first due to some red herring level issues.

In an effort to save anyone else chasing this, I basically saw this during boot of the installer:

# root device:

There was two steps to this

  1. Set hard drive mode from AHCI to IDE :(
  2. Boot from a cdrom instead of of USB stick

For reasons unclear to me booting from cdrom worked fine and successfully installed without issue. HOWEVER, booting the new installation ran into the same issue, even with the hard drive detected in either mode (IDE/AHCI), it fails to open. I never got past this and gave up on the laptop:

root device: wd0a
..
vfs_mountroot: can't open root device
cannot mount root, error = 6

Heres a lame picture to follow par for the course

barf

At this point I decided to move to the desktop, considering laptop no go. RIP

Installation - desktop

I booted off USB stick for desktop, but ran into same issue. It wouldnt detect hard drive in AHCI mode, but it did report an error. The error in dmesg which seemed to indicate hardware failure:

ahcisata0 channel 0: clearing WDCTL_RST failed for drive 0

I was very skeptical because this machine ran windows 10, Linux, other BSDs without issue right before this. I clicked it over to 'IDE' mode instead and it detected drive and started the installer. Smooth sailing from here right?

soda panic

I got a panic during auto configuration of a wifi card (which, by the way it detected no problem, go figure). I tried to reproduce it to report in GNATS but had no luck, so I just started over and skipped network configuration during installation.

At this point I had a working NetBSD 7.1.2 installation on the desktop. So at this point, I have NetBSD booting but unconfigured. I could technically move to configuration and usage at this point if I wanted. On fast hardware to boot!

Run current sources to test things

Bringing up some of the panics and really obnoxious issues, I looked into reporting/debugging them. I was told to try again on 'current' in NetBSD parlance, working sources. Which of course, is a good idea in case it has been fixed, and to ensure it is reproducible.

So I grabbed a recent snapshot of NetBSD 8-current and tried that installer. Mostly the same story there, no working AHCI.

One big thing I want to capture somewhere is that the installer failed to run due to some drivers. You'll get a panic something like this booting current:

boot growing soda panic2

After trying this tip from IRC:

14:02:38 <medfly_> you can drop to boot prompt
14:02:41 <medfly_> userconf disable i915*
14:02:44 <medfly_> userconf disable radeondrm*
14:02:47 <medfly_> userconf disable nouveau*
14:02:48 <medfly_> boot

Which by the way, the IRC channel (#NetBSD on freenode) had a lot of very patient and helpful people. The machine has an nvidia 780Ti so i try the last one, userconf disable nouveau* and boot. Bam Current installer off to the races!

I was able to install current at this point. One thing I need to figure out is how to make disabling nouveau permanent because I have to drop to boot prompt every reboot to do so. I could now use the machine. I'll probably just ssh into it moving forward.

Moving forward

After doing some cross-checking, the old intel SSD in the desktop is in fact actually starting to fail just now! This is no surprise considering its from 2008. With that in mind, I'll consider it non usable even tho it's running current doing not much right now.

I decided to just learn how to use the system in VM on my main desktop in the meantime. NetBSD mostly works great on the emulated hardware, and my primary desktop is an extremely fast machine which can build NetBSD in minutes. My beefy desktop has a 4k monitor, if you use it on 4k or similar make sure to set display scaling in virtualbox or else you will go blind:

display scaling

hello world

Eventually, I'll move some stuff around and put a good hard drive in the spare desktop and SSH into it. Now I can dig into usage, maybe run and build some software on it for starters.

Takeaway

NetBSD is very likely going to be difficult to install. Don't think amd64 will be smooth sailing just because its the comptemporary hardware of choice. While it's likely got the best coverage there are lots of open issues and lacking support for newer things like GPT/UEFI boot.

At some points I figured hacking custom u-boot code to get NetBSD to boot on a SBC would have been easier in some ways. But we'll see about that later.

Once it gets going, its pretty good. it's the consistent, rock solid UNIX experience I was hoping for. Coming from someone who primarily uses windows, but knows the basics of linux server administration, the UNIX experience equals I have no idea how to do anything in an otherwise familiar environment. I really like the simple rc.conf and rc.d init system and the sane directory hierarchies for third party software so far.

March 20, 2018

Waterboarding your brain / becoming a BSD curmudgeon

Buckle up, this is going to be a wordy, disjointed one.

“There is nothing noble in being superior to your fellow man; true nobility is being superior to your former self.”

― Ernest Hemingway

tacky, lofty quote opener: ✔️

Over the years I've been thrown into tasks at work I had zero experience or knowledge of. As those who work in the tech industry know, this isn't uncommon or even undesirable (for most).

One of the big pulls for people who are intellectually driven is the perk of getting paid to constantly tackle new and potentially interesting challenges and become better and stronger. For many, this is what makes the tech work fulfilling and satisfying. I assume this parallels in any other industry with creative or open ended efforts.

Drive and the perversions of nerdy desire

While I want to say "nobody would enjoy copy pasting mindlessly for 8 hours a day!" I worked with people doing just that for years in a few entry level jobs. For example, processing ten digit triggers in telephone switches. Another example would be 'ACDVs', Automated credit dispute verifications which consisted of copy pasting personal information from one form to blank along side it. Back when I worked in collections (over 10 years ago now..) it could not legally be automated, hence 'verification'.

boredom

Aside, let me be perfectly clear: That is completely ok. These people were not interested in becoming LLVM core contributors or porting the OCAML compiler to a MIPS processor for fun. They were working one of the only jobs in town, that they despised, to feed their family. All without complaint. And as mentioned, a few people actually enjoyed it. A certain Zen to it. I get that. I was not sitting there, stewing, thinking I was better than the work or anyone. More importantly, I was sitting right there too, grinding it out.

Barely passing high school and never going to college, I thought programming was going to probably remain just a hobby while I went through the motions at entry level jobs indefinitely unless I changed something (I didn't). And that's exactly what happened. But I practiced at home, and roughly knew what I was capable of. Getting paid to explore this further was a pipe dream. I got very lucky, That's a story for another day

The point

I want to really push myself in something I've never done before that is also technically very challenging (for me, of course). This may be a recipe for disaster, but, I'd rather face frustration and 'failure' (learning stuff on the way doesn't make this a loss) instead of becoming complacent or niche'd. Finding where your current capabilities lie and pushing them even further is about as core nerd experience as you can get.

So with that said, here's what I got in mind

UNIX and BSD operating systems

As I grow older and learn more and more about OS design, either through osmosis or casual reading, I find UNIX and modern BSD descendants growingly interesting. I couldn't tell you why, I can't run MUH GAMEZ on them. UNIX is the essence of bygone development simplicity.

unix teletype session Attracting ladies on OKCupid

The BSD family is the modern descendent of UNIX, though the family tree is quite nuanced and a story of copyright and tribulations, you can see the full thing Here (wikipedia)

I've been meaning to really get into BSD for a while, too. John Carmack having a honeymoon with OpenBSD recently really resonated with me, even just as a user of the OS.

Heck. You can run the first UNIX in docker or still read the first UNIX programmers manual. I think anyone with an interest in OS legacy owes it to themselves to get familiar or at least know about the UNIX bloodline. Yea, I'm a big hit at parties!

As I was writing this blog, I saw this tweet fly by:

https://twitter.com/BrendanEich/status/975514575012425728

I've used BSDs a little bit, but very superficially. I run FreeBSD on servers, but I couldn't tell you how to do more than install and configure PostgreSQL on one. Never mind fancy ZFS stuff.

So taking a break from complaining about there being no good video games to play, getting intimate with a BSD is probably a good deep dive choice.

Which one? Well, I was going to pick either NetBSD or OpenBSD under the impression they are still smaller than FreeBSD, whichever had a smaller USB installation media. For the current releases, by chance they are both exactly the same size.

I picked NetBSD because I find the project goals more interesting, and it seems to be in more of a need. Another one of my primary interests that I do not act on and just fetishize is embedded development. And most importantly, the deeply logical fact I like the logo.

At the end of the day, I find the folks hacking on BSDs hardcore of the hardcore (or maybe just most oldschool) of this realm, maybe unfairly so. Do I want to 'hang out with the big boys' or embarrass myself trying?

What about GNU/Linux

GNU/Linux would definitely be a more usable choice when it comes to contemporary hardware. It also has far more developers, more people using it, banging on it and writing about it too. I'm also just not as interested in it from a nerdy lore background if nothing else. I'll definitely dig into it later on. BSDs also have the advantage of the entire system being built together, userspace and kernel. I find this a huge benefit to the system expectations and stability. Not to mention no intrusive updaters or systemd (bandwagon: ✔️)

New challenge

I have some lofty goals in mind. I'd be happy to get 2 or 3 of these under my belt, especially with the way work has been going lately, I'll probably be weekend warrior'n these out. Battle plan:

  1. Install and setup NetBSD, basic usage and configuration
  2. Compile NetBSD from sources, run current/srcs
  3. Dissect boot process (mostly for extra credit real goal below)

I really haven't done any of the above, but this is all straight forward and documented. There are a lot of other minor tasks this involves. For example, The BSD projects still use CVS (for some good reason I presume), so that would be another thing to learn. These will all probably not be a challenge, just prerequisites. Hell, I should admit, I've never done any sizable development in C, so I'll need to brush up on that (along with compiler extensions, buildchain etc).

Here is a nice NetBSD milestone:

bring up NetBSD on a new or unsupported SBC/board

How cool would that be? It'd require a good understanding of the hardware AND the software from ground zero if it was a new board.

I have a lot of single board computers, a few of which are newer, like the Libre Computer and ASUS tinker board, surely I can find one that isn't currently supported. Of course I have Raspberry PIs too, but I'm sure lots of others will try to get those going (not that I'm opposed to helping out).

They are both ARM processors, so I don't think it would be a soul crushing endeavor. I'll likely just pick one and go until I get things booting (if no one else has), I'd prefer the Libre computer (Amlogic ARM Cortex A-53), I think the Rockchip ARM Cotrex-A17 the ASUS Tinker board has more coverage.

If I wanted to go an extra step, I could even try to do so for a MIPS board, the MIPS Creator CI20 from imgtec. It's basically dead in the water and they've sold back MIPS, probably not the most fruitful. In fact, one of the NetBSD devs started the process years ago.

I'm also, for the first time in my life, well equipped to tackle early boot (well, as much as you can be) debugging. I own a logic analyzer, several USB FTDI serial cables, an oscilloscope, etc. But this would be more for new board bringup. Slightly different ARM SoCs is probably not as bad.

Yep, that's it! How hard could it be right? How long before I lose interest and just stop talking about it like all those other projects? Only time will tell. If I could even finish the first 3 things, that'd be a net win for me (no pun intended). This post is already far too long so I shall document the journey in more discreet chunks! Instead of a glacier of content!

<titanic picture here>
March 17, 2018

MSP430 assembly part 1: Blinking LED hello world

Lets blink an LED on a MSP430 launchpad the old fashioned way! Assembly! Don't worry, MSP430 has a nice, small RISC instruction set. It's only true 24 instructions, with a total of 27 higher level ones emulated by the assembler.

The full user's guide for the MSP430 microcontroller i will be covering (on an older launchpad) is available here ,
grab a copy because it covers the instruction set and microcontroller completely.

First, I will present the source with the CCS generated boiler plate for ELF sections and stack setup:

            .cdecls C,LIST,"msp430.h"       ; Include device header file
            .def    RESET                   ; Export program entry-point to
                                            ; make it known to linker.
            .text                           ; Assemble into program memory.
            .retain                         ; Override ELF conditional linking
                                            ; and retain current section.
            .retainrefs                     ; And retain any sections that have
                                            ; references to current section.

    RESET       mov.w   #__STACK_END,SP         ; Initialize stackpointer
    StopWDT     mov.w   #WDTPW|WDTHOLD,&amp;WDTCTL  ; Stop watchdog timer

            ; our code will go here moving forward

    .global __STACK_END
    .sect   .stack
    .sect   ".reset"                ; MSP430 RESET Vector
    .short  RESET
            

For all intents and purposes we can ignore the setup scaffolding, it just turns off the watchdog timer and setups the stack and reset vector. This lets you press the reset button to .. jump to our RESET label.

To blink an LED on a microcontroller you need to do 2 things:

On the Ti launchpad, there are two LEDs on PORT1. For simplicity we can set the entire port to output. To do this, mov an immediate value into the named mapping like so:

mov #0FFh, P1DIR ; P1DIR is port1 direction control. '#0FFh' is an immediate value in hex

Now the entire port is set as output. To check, you could just try to lightup all the LEDs by turning all the pins on:

mov #0FFh, P1OUT

However this isnt very exciting because they just stay lit. We need to toggle. The easiest way is an exclusive or (xor) of the bits.

xor #0FFh, P1OUT  ; flip all bits in P1OUT

However, we need to loop over this so we dont just change the output once and fall through:

LOOP
    xor #0FFh, P1OUT  ; flip all bits in P1OUT
    jmp LOOP
    

Astute readers know what's coming next. if you run this code the LEDs will appear to be solid. It's because the loop's executing so fast the human eye cannot see the difference. In fact there probably is no difference, as the rate this loop executes is many 1000s of instructions per second.

I lied when I said we only need to do 2 things. We need a delay loop. Unfortunately we do not have something as convenient as wiring's delay(). We need to make a delay loop by hand! This is much easier than it sounds. There are a few ways to do so, we will do the naive approach first:

The tricky part about this approach is figuring out how big of a number you need, which can depend on the MCU clock rate, and what instructions you effectively execute within the loop. Precise timing can be achieved with some math and nop sleds. But for our purpose, we can ballpark some visible delay.

Lets stuff our countdown in the first general register R4. The MSP430 is a 16-bit cpu so lets try the biggest 16-bit unsigned value to start with:

mov #0FFFFh, R4

Now lets run it down to zero. We can check if it hit zero using the status register SR, this is like the x86 EFLAGS register. It will have bits set automatically after certain instructions, which are documented for every instruction that affects status flags. Alternatively, there is a conditional jump when the zero flag is set, so we will use it - jnz.

DELAYLOOP
    SUB #1h, R4 
    jnz DELAYLOOP
    
    ; we fall through here once delay loop is done 
    

This will spinwait until R4 hits zero and then fall through, now all that's left is to jump to top loop again:

    jmp LOOP 
    

This will start the whole process over. On a MSP430G2553, this first try at a delay loop works out to around 250ms, making for a nice blinking rate.

For completeness, the entire body of our effective code looks like this (with some added comments):

            mov #0FFh, P1DIR    ; setup - set port1 to output
LOOP
            xor #0FFh, P1OUT    ; flip port 1 bits
            mov #0FFFFh, R4     ; R4 will be our delay counter

DELAYLOOP
            SUB #1h, R4         ; subtract 1 from R4...
            jnz DELAYLOOP       ; if we hit zero we're done with delay loop
            jmp LOOP
            

This code can be simplified even further, do you see how? This is an exercise left to the reader. It's also midnight and I'm starting to get tired in my old age.

In part two we will use the built in timer peripherals of the microcontroller and use interrupts for a cleaner approach. After that it will be on to PWM (Pulse Width Modulation) duty cycles and controlling a servo using an ADC input.

March 12, 2018 · asm · embedded · msp430 · blinky · assembly

Simple sampling with Box-Muller transforms pt 1

I'm a technically uneducated idiot, so talking about math is a bit above my pay grade, but Box-Muller transform is a straightforward, computationally simple way to generate a distribution of points.

This is useful in many cases where you need to generate samples for simulations, so I think every programmer should have familiarity with it.

Lets take the example code from the wikipedia and convert it to standard C. This program will just print 60 x/y samples.

First, checkout the cool visualization of the transform on uniform inputs from the wikipedia! If you right click and view the image directly, you can hover on the crosses to see points there.

#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <float.h>
#include <time.h>

static const double epilson = DBL_MIN;
static const double pi2 = 2.0 * M_PI;

struct sample {
    double z0;
    double z1;
};

struct sample box_muller_sample() {
    double u1;
    double u2;
    struct sample ret;

    do {
        u1 = rand() * (1.0 / RAND_MAX);
        u2 = rand() * (1.0 / RAND_MAX);
    } while (u1 <= epilson);

    ret.z0 = sqrt(-2.0 * log(u1)) * cos(pi2 * u2);
    ret.z1 = sqrt(-2.0 * log(u1)) * sin(pi2 * u2);
    /* no sigma or mu */
    return ret;
}

int main(int argc, char **argv) {
    time_t curtime;
    srand(time(&amp;curtime));

    puts("Box-Muller samples:");
    for (int line = 0; line < 10; line++) {
        for (int i = 0; i < 6; i++) {
            struct sample bm = box_muller_sample();
            printf("%0.3f:%0.3f\t", bm.z0, bm.z1);
        }
        printf("\n");
    }
    return 0;
}

a sample output looks like this:

Box-Muller samples:
-0.559:0.786    -0.798:-0.640   -0.365:-0.140   1.417:-0.197    0.210:1.734     0.738:1.026
-2.334:0.079    -0.973:-0.634   -0.674:1.263    0.765:1.705     -0.560:0.860    -0.576:-0.044
-1.242:-1.705   1.705:-0.471    0.063:-0.079    0.732:-1.173    -0.242:-0.756   -1.798:0.853
-1.088:1.138    2.096:1.439     0.556:0.168     -0.105:0.132    -0.397:-0.728   0.090:0.109
-0.850:-0.034   0.930:0.108     -1.105:0.016    -0.592:-1.795   1.073:-0.227    -1.198:-0.517
-0.901:-1.978   -0.524:-0.090   0.821:-1.595    0.783:-0.051    0.580:-0.458    0.416:1.115
0.888:0.337     -0.074:-0.526   1.667:0.233     -1.339:0.075    0.563:-0.379    0.236:-0.229
0.222:0.472     -0.364:0.061    0.413:1.185     0.265:0.083     -1.120:1.978    0.939:0.807
-0.344:1.586    0.162:-0.274    -0.309:-0.098   -0.728:-0.441   -0.358:-0.310   1.324:-0.816
0.187:-0.160    0.035:0.083     1.386:-0.823    0.117:-1.897    0.475:-0.387    0.295:-0.382

In a future post I'll cover an even more efficient algorithm to generate uniform sampling, which is appropriate when a very large amount of random sampling numbers is needed. It's called Ziggurat

March 8, 2018 · c · sampling · math

Creating multiboot ELF kernels with FASM

I've been meaning to write about this one for a while, since I hacked together some simple OSes last summer! Multiboot is magical.

Multiboot is a specification that provides a standard format for bootloaders to load kernels.

The most common (and in fact only I know of) implementation is unsurprisingly GRUB. There is actually two multiboot specifications at this time:

I will cover creating a multiboot 1 loadable 32 bit kernel using FASM. The killer nice thing is, you can boot an ELF directly with Multiboot.

Prerequisites

A basic knowledge of ELF layout is strongly recommended. If you are not familiar with ELF format layout sections, this may be a bit foreign, I recommend at least skimming an overview of the sections, here is a good overview. Ange Albertini (of corkami.com) has amazing poster graphics as well here.

Creating an ELF with FASM

I have to admit the assembler I remember how to use well enough anymore is FASM, so I will cover how to generate a bootable elf with it. It is probably easy enough to translate mnemonics to nasm or gas if you have the docs handy. One benefit of using FASM is it handles linkage for you so you can assemble a working binary in a single step. However, NASM is more popular.

Our first step is to generate an ELF, lets do that first. This will make a buildable empty ELF:

format elf ; you can use elf or binary with mb kludge
org 0x100000 ; this will be your kernel reserved memory
use32 ; 32 bit

_start:
; we will add multiboot header here!

_kstart:
; kernel here

; data here
_end_data:
; bss section here

; reserve bytes for kernel stack
rb 16384
_kstack:

_end:

Check your work pt 1

Save this file as kernel1.asm Make sure you didnt typo it by trying to compile it:

$ fasm kernel1.asm
flat assembler  version 1.71.62  (1048576 kilobytes memory)
2 passes, 16683 bytes.

If you get no errors (and a kernel1.o) it worked. We can now move on to adding a multiboot header!

Header magic

The only thing you have to do for a binary to be loadable is include a correctly formatted header (and add it to your grub list). Importantly you do not need to fill out the complete header. I've recreated the multiboot 1 header here:

Offset Type Field name Note
0 u32 magic required
4 u32 flags required
8 u32 checksum required
12 u32 header_addr present if flags[16] set
16 u32 load_addr present if flags[16] set
20 u32 load_end_addr present if flags[16] set
24 u32 bss_end_addr present if flags[16] set
28 u32 entry_addr present if flags[16] set
32 u32 mode_type present if flags[2] set
36 u32 width present if flags[2] set
40 u32 height present if flags[2] set
44 u32 depth present if flags[2] set

You can see the multiboot1 docs here, as well as full latest specification, scroll to 'Boot information format'

The idea is to setup magic, flags, checksum correctly. You toggle certain flags to have multiboot fill out fields you are interested in and let it know what it is loading.

Adding multiboot header

To make the ELF we created above bootable, we need to fill out the multiboot header. It needs to be the first thing in the binary the multiboot loader will see. This means we need to stick it directly under _start and that is why there is _start and _kstart.

I will present the entire filled out header below. Add it below the _start label we created before. Save this file as kernel2.asm to keep track of our work.

; ... snip

_start:
; this is the multiboot header
mbflags=0x03 or (1 shl 16)
dd 0x1BADB002
dd mbflags   ; 4k alignment, provide meminfo
dd -0x1BADB002-mbflags  ; mb checksum
    dd _start       ; header_addr
    dd _start       ; load_addr
    dd _end_data    ; load_end_addr
    dd _end         ; bss_end_addr
    dd _kstart      ; entry point
; end mb header

Here is an explanation of what each line is accomplishing

mbflags=0x03 or (1 shl 16)

This is taking 0x03 and bitwise or'ing it with (1 << 16). 1 shifted left 16 is 65536.

in other words It is taking these two binary values (shown as 16 bit/u16 since thats all we need):

(Note: this is BIG ENDIAN / logical)

0x03: 1100 0000 0000 0000
(1 shl 16): 0000 0000 0000 0001

bitwise or'd: 1100 0000 0000 0001

Following the multiboot docs,

we have set the following bits:

The big gotcha for me said bit 16 is not required for ELF format kernels:

This information does not need to be provided if the kernel image is in elf format, but it must be provided if the images is in a.out format or in some other format.
(which is known in sources as `MULTIBOOT_AOUT_KLUDGE`)

I always had to provide it even with the above ELF, Im unsure if I built the sections incorrectly or otherwise, but all online sources I could find did same when building ELF kernels.

We will put these flags in header later on.

dd 0x1BADB002

this satisifes the 'magic' part of the header for multiboot1

dd mbflags

Put our flags we set above in the right spot

dd -0x1BADB002-mbflags

This is a tricky way to set the checksum of the 3 required fields which needs to be 0 mod 2^32.

I will breeze over the rest, loading our _start,_end_data, _end and _kstart addresses into the header with dd lets multiboot our elf sections and where to jump to after loading.

That's it (phew)!

Lets a hlt in _kstart:

_kstart:
hlt

This gives an effective memory address in our kstart label.

Check your work pt 2

Make sure we can assemble:

$ fasm kernel2.asm
flat assembler  version 1.71.62  (1048576 kilobytes memory)
2 passes, 16683 bytes.

In theory, this kernel2.o is actually bootable but we won't be able to tell because it will hang with the QEMU boot messages still visible.

Add some video functionality

Lets add some basic video memory functionality, I will not cover this in detail, I will provide working code, but its an exercise for the reader.

It will clear the framebuffer and write a message, confirming we did actually load and jump to our _kstart code.

here is the full listing with a hello world:

; fasm multiboot example
; this shows how to use elf (or bin if you want)
; fasm output with a multiboot header with grub

format elf ; you can use elf or binary with mb kludge
org 0x100000
use32

_start:
; this is the multiboot header
mbflags=0x03 or (1 shl 16)
dd 0x1BADB002
dd mbflags   ; 4k alignment, provide meminfo
dd -0x1BADB002-mbflags  ; mb checksum
    dd _start       ; header_addr
    dd _start       ; load_addr
    dd _end_data    ; load_end_addr
    dd _end         ; bss_end_addr
    dd _kstart      ; entry point
; end mb header

; code
_kstart:
    ; set stack right away
    mov esp, _kstack

    ; grub sets up 80x25 mode for us
    mov edi, 0xB8000 ; video memory

    ; the screen data is left as is, showing
    ; qemu boot messages and crap, so clear it out

    ; since we dont care about the actual
    ; rows and heights we can just linearly nuke the total
    ; num of bytes:
    ; 80x25 = 2000 chars,
    ; each visible char has a value byte and display control byte
    ; so total bytes = 2000 * 2, 4000 bytes
    ; however we can simplify this by setting a full 32 bits each
    ; loop (4 bytes or 2 chars)
    mov ecx, 1000
    cld
    @@:
        ; set 1F control bytes, 0x00 text bytes
        mov dword [edi + ecx * 4], 0x1F001F00
    loop @b

    ; now display a message before halting
    mov esi,msg
    mov ecx,msglen
    @@:
        lodsb
        stosb
        mov byte [edi], 0x1F
        inc edi
    loop @b

    hlt

; data section
msg db 'hello from a multiboot elf'
msglen = $ - msg

_end_data:

; bss uninit data here


; reserve the number of bytes of how big you want the kernel stack to be
rb 16384
_kstack:


_end:

Save it as kernel3.asm

Lets assemble and boot it!

$ fasm kernel3.asm
flat assembler  version 1.71.62  (1048576 kilobytes memory)
2 passes, 16683 bytes.
$ qemu-system-i386 -kernel kernel3.o

Amazing!

multiboot

If you got stuck, full sources and a script to make bootable as an ISO using el-torito (covered later on) is available on my github

next steps and other resources

Understanding writing to the video memory above: https://wiki.osdev.org/Printing_To_Screen

NASM version

February 25, 2018 · os · bootloader · assembly · elf

easier updates

I found a much easier way to blog without messing about, blot.im. This is great! Not that my old setup was hard (static site gen + upload to s3), but anything easier is a welcome change.

Anyway with that said, I'm going to start blogging smaller, more frequent 'nuggets' of things as I think of them. I dont wan't to call it micro-blogging, but something bigger than twitter and something you wouldnt expect to see on a blogspot.

In my day to day work I tend to come across lots of tricky stuff I may or may not lose in the annals of time so this will by my efforts to combat that while sharing them at the same time.

So expect more succinct, raw development notes, and bits here and there.

February 11, 2018 · site · meta

dexGame engine boogaloo

Well it finally happened, I threw in the towel on the current engine tech I was using for dexGame / BAA. I will detail why, but first some background as to how I ended up here.

Background

I was looking to start a simple, ruthlessly barbaric shooter game. I didn't need triple A features or crazy artist workflows. However, I did want responsive, good feeling input for a fast shooter. Here are the candidates I seriously considered:

Unity: Amazingly, I didn't pick unity, because I had an unfair, dated stigma about it's input handling and overall 'feel' for fast shooters. My main concern was hitting a point where I wish I had the sources to fix an issue, or customize input or movements. It's definitely back up on the chopping block. If nothing else, I know C# very well, it was my bread and butter professional development language for over 5 years.

Unreal: While I like unreal and have made throw away project and demos with it, I had a hankering to use something more 'lightweight' in terms of code/cognitive load for a fresh game. If I was going to go for something with full code, I'd prefer something where I could understand the majority of the engine front to back. I'm not afraid to admit Unreal is beyond my mental capacity to even pretend to understand each subsystem. Aside from that, I could implement the game in blueprints if I really wanted, but I kept looking.

Lumberyard: I ended up at lumberyard trying to figure out what the heck happened to the old crytek sdk licensing. I like what Amazon is doing with it (tho the AWS integration is lukewarm for really fast shooter servers). I strongly considered this due the amount of work and money Amazon is dumping into it but I was put off by a few things,

With that in mind I kept looking.

Tombstone: This engine, previously known as C4, is a one man show. Written by Eric Lengyel (along with many other tools written from scratch for the entire pipeline), It boasts some interesting features and a completely integrated editor. Importantly, he heavily sells his code quality and architecture on the website. All of the model & map, sound, video, texture formats are completely homegrown here. However, he also developed an open exchange format that no one else uses. Some island syndrome here.

Old quake engines etc: I have a lot of experience hacking on quake2, quake3 and source engines, so it was tempting to use something like that but a few issues I had with going this route include:

It's also worth noting valve doesn't care about providing SDKs for modders at all anymore (can you blame them?), after a 5 year hiatus the source2013 sdk was dumped and left for dead and all the tooling is held together with duck tape.

Initial Decision

Ultimately, I drank the kool-aid and went with Tombstone. If I had full sources, I wanted to have a hope of understanding most of the interesting systems and I have to say, Eric is not exaggerating. The code quality is absolutely excellent. It's written in modern C++ and it was the most readable code of any non-trivial system in any language I've ever seen (while still being C++) which says a lot! For my goal of understand the code (well most of the important system) I succeeded thanks to the excellent engineering.

The systems make sense and work well together. It's not free, but for what you get, with no royalties or runtime fees it is a serious bargain.

The content system is a little different, but you can actually create models and maps within the integrated editor, as well as 2D gui systems which was extremely handy. Furthermore, my game compiled and shipped was under 6MB total. It's a very sweet package and compiles nicely (well mostly, linux require minor opengl header fuss)

Breaking point

If I am praising Tombstone so much, why am I moving away from it? While it was great, overall several small things slowly grated at my already small chances of completing a one man game. None of these are critical, but over time it made me start to consider more and more

Not to sound like I'm picking on this engine, I really do like it and still highly recommend it if you can work within those constraints. If I was making a small single player game or visualization/simulation software I'd very likely pick it up again.

Moving forward

Ill probably pick something that starts with U unless lumberyard really makes impressive strides

January 28, 2018 · dexGame · engine

Happy holidays

Hope everyone had a good christmas and a relaxing break. There will be a bit of a lapse of updates while I find something lower friction for posting updates

December 27, 2017

View the archives