Author: admin

Kernel Audit Milestone

Kernel audit tool now reports 67% new code meaning now less than 1/3 of the kernel is still based on legacy code.

This milestone coincides with most of the system call API now having been rewritten (but still heavily depending on some old and partly updated functions internally).

System Calls Need More Fixes

The current API still mostly replicates the old one with a few changes only where necessary, this makes testing across updates a little easier for now.

Future versions will improve details such as using 64-bit sizes/offsets/indices where applicable. Proper system call documentation will follow as I start to prepare a final “1.0” API.

Coming Up

• Build system improvements especially for dealing with different hardware configurations
• Scheduler & threading improvements
• Further work on container support & maybe some prep for hardware virtualisation
• More work towards proper hardware drivers

I’m having a lot of troubles with the local authorities and a couple of issues with things like uploading photos on the new site (works ok from my development machines but not phone…), but in good news I’m still managing to get new test hardware from China.

A couple of models of motors arrived this last week, I’ve got a surplus of boards for doing more extended testing once I can get a proper workbench set up somewhere, and I’ve already paid early for a discount on a new high performance board (the Titan) I’ll hopefully be ordering very soon. (UPDATE: Okay maybe the preorder vouchers were cancelled? Waiting to hear back from the reseller about what’s going on with that launch.)

I’ve now written some replacement code for PLIC (the interrupt controller on RISC-V) as well as some new code for toggling GPIO pins but this is left to be tested & finished another day.

A full device driver kit will come later but I figured access to GPIO pins would be the most useful driver overall! I’ve found a bit of reference information for GPIO on a couple of devices and I’ve only implemented limited support for one so far but it should be straightforward to get some GPIO going on different devices as long as I can find enough information.

I’ll need to decide on some API for GPIO access before this will be usable from user programs or from other drivers. The other important part needed to get some drivers working is some mechanism to configure devices from the device tree info, at the moment the system parses the file but doesn’t do much with the information except for detecting memory, so it will at least have to pass on some base addresses or other info to individual drivers.

Increased testing has identified a number of problems and is starting to lead to solutions. This post will go over the status of some recent changes as well as test results.

Filesystem

The filesystem update is now almost complete, with only around 400 lines of legacy code remaining to be replaced in filesystem syscall functions.

Testing revealed one major stability issue resulting in inode table exhaustion, this was a simple reference counting error and has been fixed.

Overall the filesystem code is mostly stable for simple deployments but extensions like support for large disks are still in progress. The filesystem code supports using filesystems from earlier versions, allowing for more stable & more full featured options.

Some VFS-like support also exists by way of a “drives” structure for multiplexing filesystems, which is designed to work with the container system, but this will be discussed in another post.

Compiler

Testing also revealed some problems when using the filesystem code with the new compiler backend, these have partly been worked around with rewrites but some compiler bugs are still affecting the filesystem code.

The compiler is likely to still have some known bugs at release, so will initially be a demo together with the new system (building the system with GCC will be more stable for the near future). Once features are finished more compiler testing will be enabled hopefully resulting in a very stable platform down the track.

Scheduler

A few tests have been crashing the scheduler, which sounds bad, but I have already written a new fallback scheduler implementation which should address any such stability issues while the main scheduler implementation is improved.

So as the kernel code stabilises there should actually be separate “stable” and “performance” scheduler options, and hopefully these will be usable side-by-side on the same system.

Container Security

I’ve avoided adding legacy security features like user/group support or fine-grained capabilities in favour of native support for containers. This should make for easier real-world security as the container model can easily be adapted for multi-user and multi-app security barriers, and this model fits the design of the rest of the system.

Container support half-exists in the kernel already so some things already go through the container layer, but proper management of containers & other security features won’t be fully enabled until later. Due to the system being designed around this model it should be fairly robust relative to early container implementations, with a different set of core kernel structures for each “container” and possibly eventually some address space isolation between them too (not yet implemented).

Hardware Ports

Testing on real hardware has finally commenced but has been slightly delayed by the need more more testing of core features (which I’ve been doing in an emulator), so I hope to get back to that shortly.

Initial testing has been done on the Ky X1 CPU with memory & multicore support fully or almost fully working as of the last tests but no hardware drivers for testing any real stuff (only a serial console & RAM disk). This work is expected to also be applicable for booting on other RISC-V 64 devices but I don’t know how similar other driver needs will be across devices yet.

Once I can get back to hardware work I hope to at least get GPIO drivers working on one or two boards and add more interrupt handling support so that drivers for simple robotics devices can start being implemented.

Networking & Graphics

Some work has been done on these subsystems at least in planning but probably won’t be fully enabled at release. Some demos may be possible soon, don’t hold your breath.

Kernel Audit

The kernel is currently around 64% new code. Old code continues to be replaced, however early development also swelled the original codebase a little leaving currently just under 5,000 lines of “old” code still left to replace.

The filesystem code is now almost entirely new and supports some extended format versions to start testing more heavy duty workloads. The few bits of old filesystem-related code are being steadily replaced now.

The scheduler code is partly new but leans heavily on old functions in proc.c, further scheduler improvements will probably be synchronised with the container support.

The rest of the kernel is about half new/half old with things like support for device tree blobs added recently but some other startup code & utility functions being old. Some unused or redundant code is still laying around (such as currently having two copies of some page table functions), so the code statistics are a little displaced by such things but represent the in-development codebase.

Userland & Tests

Some work has been done on new “userland” programs & ports, so testing has also been done with some almost-real-world programs but this will be covered in another post. The “libc” implementation is almost entirely my own code, it’s missing many functions but otherwise is working quite well.

The old test suite itself is still not completely working so around half the tests are still disabled, but support is improving. An additional test also exists for multithreading, which is partly stable and will probably be enabled at release but with limited functionality.

Summary

With the filesystem code relatively stable and stability of the scheduler improving I seem to be on track to deliver a stable and high performance core OS, but of course am behind schedule and need to finish many things.

Some progress has been made porting the test suite from xv6 to the new compiler backend & updated kernel.

One serious stability bug was discovered in the filesystem code, with the system running out of inodes after some number of runs of the test suite (meaning files aren’t being closed properly somewhere…), and some bugs in the compiler are also exposed by the test suite. About half of the original tests are currently enabled, with some additional bugs to investigate in the others (while some have just been disabled due to being too slow or configuration-sensitive).

Will try to return to code!

(If you can’t tell!)

I’m aware of some minor problems with the site like menus not looking good on mobile and missing contact forms, I’ll try to fix these eventually but will be prioritising other things.

In the long term hopefully I’ll be able to hire someone to maintain the site but for now I have a lot of things to balance and will probably have to accept a few annoyances with site maintenance to avoid losing too much time tweaking CSS.

If you check back in a month or so I should have the site more prepared, in the meantime I’ll try to post more updates, write some better introduction pages and ready some actual code releases. Your patience is appreciated!

Aside from coming up with ideas for “release engineering” there are a few parts of the OS which are currently under heavy development and may or may not be ready in time for the first proper releases:

PLIC & Driver Kit

The system currently only supports some ad-hoc hardware drivers for testing, but does support loading Device Tree Blob files so can be extended to support real hardware drivers with a bit more work:

• PLIC & interrupt handling code needs to be extended or rewritten to allow dynamic configuration from the DTB
• Management of modules & drivers is not fully fleshed out yet, but is likely to work similarly to other systems (e.g. “device files” are still used currently with limited functionality but could be streamlined into a more modern configuration)
• Alternative filesystems, advanced disk management and partitioning support will probably be delayed until after some releases or released with limited functionality (prioritising enhancements to the core filesystem instead to ensure it remains stable)
• Framebuffer graphics can easily be added for some (emulator) targets but will likely be more problematic on real hardware
• No work has commenced on any support for device hub technologies such as PCI & USB, support for legacy device hubs will probably be limited for a little while in favour of drivers for smaller individual devices & GPIO

Full Self-Hosting

The system is already quite stable when built with the custom compiler, but more work is needed especially on the assembler/linker and at the build system end in order for the OS to be able to fully recompile itself.

In the meantime most development is done from Linux, which is a reasonable compromise and is the same way most other operating systems are developed, but the system is fairly close to being self-hosted and this is expected to go very smoothly once the new linker rewrite is finished.

Networking Support

I’ve been looking into a few different options to provide networking support, none of these options are perfect so the final system may come with a couple of alternatives but these probably won’t be ready in initial releases:

• Writing a new TCP/IP stack – this is likely to be viable for small demos but is unlikely to result in a competitive system in the short term
• Porting an existing TCP/IP stack – this is likely to be much more practical but will probably be limited to some typical “embedded networking” feature set and perhaps a bit clunky to set up for end users
• Developing new protocols – this will likely result in a much better demonstration systems and perhaps will be more useful for fully integrated products (especially for HPC & robotics) but won’t integrate nicely with legacy tech
• Running full networking using virtualised/emulated systems – this will probably be a good compromise for full featured networking in the mid term but won’t solve short or long term integration problems

For the time being, some legacy internet APIs including kqueue/kevent functions are already exposed in the C library but aren’t implemented in the OS, allowing some networked programs to be built or prototyped using legacy APIs but they won’t run on the new kernel yet.

Documentation & End User Utilities

Documentation is currently limited and no man (manual page) program is included. This will be addressed as individual components stabilise, in the meantime you will need to check instructions printed by the individual programs or written in the source code.

Some other utilities needed by users but not critical for development will also have been overlooked at this stage, but more programs can easily be ported later.

During the build process for the kernel, a program just scans code for markings “NEW CODE” or “OLD CODE” in each file and prints a report. Obviously this simple method can be adapted to check for more complicated audit requirements e.g. checking that each file has been reviewed within a certain timeframe (the current process just checks which files have been written anew versus which are old code).

Totals are rounded so will usually add up to just under 100%. This is only a simple tool and can be extended later for more thorough audit statistics!

But this is a really powerful tool. You ever written an OS kernel? I’ve written most of one, but I did it the easy way: One piece at a time.

// Copyright (c) 2025, Rainbow (Zak) Yani Star Fenton
// This "audit" program is licensed under the Mulan PSL v2. You can use this
// software according to the terms and conditions of the Mulan PSL v2.
// You may obtain a copy of Mulan PSL v2 at:
// http://license.coscl.org.cn/MulanPSL2
// THIS SOFTWARE IS PROVIDED ON AN “AS IS” BASIS, WITHOUT warranties of
// any kind, either express or implied, including but not limited to
// non-infringement, merchantability or fit for a particular purpose.
// See the Mulan PSL v2 for more details.

#include <stdlib.h>
#include <stddef.h>
#include <stdio.h>

#define AUDIT_TYPE_OLD      1
#define AUDIT_TYPE_NEW      2
#define AUDIT_TYPE_UNMARKED 3

typedef struct audit_info audit_info_t;
typedef struct audit_filelist audit_filelist_t;
struct audit_filelist {
  const char* name;
  audit_filelist_t* next;
  int type;
  int lines;
};
struct audit_info {
  int filecount;
  int linecount;
  audit_filelist_t* oldcode;
  audit_filelist_t* newcode;
  audit_filelist_t* unmarkedcode;
};

int audit_filelist_countfiles(audit_filelist_t* l) {
  int total = 0;
  while (l) {
    total++;
    l = l->next;
  }
  return total;
}

int audit_filelist_countlines(audit_filelist_t* l) {
  int total = 0;
  while (l) {
    total += l->lines;
    l = l->next;
  }
  return total;
}

int usage(int argc, char** argv, const char* error) {
  FILE* out = error ? stderr : stdout;
  fprintf(out, "ABOUT:\n");
  fprintf(out, "This is a simple code audit tool to note occurrences like OLD CODE & NEW CODE\n");
  fprintf(out, "The main purpose of this tool is to help gradually remove old/third-party code from a codebase,\n");
  fprintf(out, "but it could be extended later e.g. for checking each file has been manually audited at a recent date.\n");
  fprintf(out, "This tool isn't specific to a particular programming language and can be used for config files too.\n");
  fprintf(out, "\nUSAGE:\n");
  fprintf(out, "\t%s [source files...]\n", argv[0]);
  if (error) {
    fprintf(out, "ERROR: %s\n", error);
    return -1;
  } else {
    return 0;
  }
}

int audit_check(const char* line, const char* check) {
  char c;
  while ((c = *line++) != 0) {
    if (c == check[0]) {
      const char* search = check+1;
      while (*search && *search == *line) {
        line++;
        search++;
      }
      if (!*search) {
        return 1; // Found!
      }
    }
  }
  return 0; // Not found
}

void audit_line(audit_info_t* info, audit_filelist_t* file, const char* line) {
  if (audit_check(line, "OLD CODE")) {
    file->type = AUDIT_TYPE_OLD;
  } else if (file->type == AUDIT_TYPE_UNMARKED && audit_check(line, "NEW CODE")) {
    file->type = AUDIT_TYPE_NEW;
  }
  file->lines++;
}

char linebuf[1000];

int audit_run(audit_info_t* info, const char* filename) {
  FILE* f = fopen(filename, "r");
  if (!f) {
    return -1;
  }
  audit_filelist_t* filelist = malloc(sizeof(audit_filelist_t));
  if (!filelist) {
    fprintf(stderr, "ERROR: Failed to allocate memory\n");
    fclose(f);
    return -1;
  }
  filelist->name = strdup(filename);
  filelist->type = AUDIT_TYPE_UNMARKED;
  filelist->lines = 0;
  filelist->next = NULL;
  
  while (fgets(linebuf, 1000, f)) {
    audit_line(info, filelist, linebuf);
  }
  
  switch (filelist->type) {
    case AUDIT_TYPE_NEW:
      filelist->next = info->newcode;
      info->newcode = filelist;
      break;
    case AUDIT_TYPE_OLD:
      filelist->next = info->oldcode;
      info->oldcode = filelist;
      break;
    default:
      filelist->next = info->unmarkedcode;
      info->unmarkedcode = filelist;
      break;
  }
  info->linecount += filelist->lines;
  
  fclose(f);
  return 0;
}

void audit_subreport(audit_info_t* info, const char* startline, audit_filelist_t* files, FILE* out, int extrastats) {
  int nfiles = audit_filelist_countfiles(files);
  int nlines = audit_filelist_countlines(files);
  int fpercent = (int) (((double) nfiles) / ((double) (info->filecount)) * 100);
  int lpercent = (int) (((double) nlines) / ((double) (info->linecount)) * 100);
  fprintf(out, "%s %d FILES (%d%%)\t%d LINES (%d%%)\n", startline, nfiles, fpercent, nlines, lpercent);
  if (!files) {
    return;
  }
  //fprintf(out, "%s IN ", startline);
  audit_filelist_t* smallest = files;
  audit_filelist_t* largest = files;
  while (files) {
    //fprintf(out, " %s", files->name);
    if (files->lines < smallest->lines) {
      smallest = files;
    } else if (files->lines > largest->lines) {
      largest = files;
    }
    files = files->next;
  }
  //fprintf(out, "\n");
  if (extrastats) {
    fprintf(out, "%s SMALLEST %s (%d lines) LARGEST %s (%d lines)\n", startline, smallest->name, smallest->lines, largest->name, largest->lines);
  }
}

void audit_report(audit_info_t* info, FILE* out) {
  fprintf(out, "THIS IS AN AUTOGENERATED REPORT\n");
  fprintf(out, "This report was created by the 'audit' program\n");
  fprintf(out, "TOTAL %d FILES %d LINES\n", info->filecount, info->linecount);
  audit_subreport(info, "NEW CODE:     ", info->newcode, out, 0);
  audit_subreport(info, "OLD CODE:     ", info->oldcode, out, 1);
  audit_subreport(info, "UNMARKED CODE:", info->unmarkedcode, out, 1);
}

int main(int argc, char** argv) {
  audit_info_t info;
  info.filecount = 0;
  info.oldcode = NULL;
  info.newcode = NULL;
  info.unmarkedcode = NULL;
  
  for (int i = 1; i < argc; i++) {
    if (audit_run(&info, argv[i]) != 0) {
      fprintf(stderr, "ERROR: Failed to process file '%s'\n", argv[i]);
      return -1;
    }
    info.filecount++;
  }
  
  if (info.filecount < 1) {
    return usage(argc, argv, "Expecting list of source files\n");
  } else {
    audit_report(&info, stdout);
    return 0;
  }
}

I messed up my original logo and domain but I’m back!