ToaruOS - 2018 Year in Review
ToaruOS - 2018 Year in Review
K. Lange <klange@toaruos.org>
The year is coming to a close, which means it's time to look back on what has
happened over the past twelve months in the world of ToaruOS. This was a very
big year for us, seeing the commencement and conclusion of the NIH project.
We'll take a deep dive into the work that went into that project, as well as
the efforts to bring the ports catalogue back to where it was previously. We'll
also look at the new features of the OS, both in kernel and userspace. Finally,
we'll wrap things up with a look forward to 2019 and the future of ToaruOS.
The NIH Project #
For the first six years of its life, ToaruOS employed Newlib as a C standard library. When I started on ToaruOS as a university student, building everything from scratch wasn't a goal. ToaruOS made liberal use of third-party libraries in its userspace, particularly Cairo and Freetype, so the use of a third-party libc never particularly bothered me in those early years. As ToaruOS developed, though, Newlib became a detriment to progress. Adding new functionality that wasn't already present in Newlib meant adding more hacks to the Newlib headers, and further expanding an already bloated glue layer.
When I set out to replace Newlib in March, the idea of also dropping Cairo as a requirement for the compositor surfaced. ToaruOS already had a reasonably robust graphics library which was capable of providing rendering for the compositor, but it was painfully slow. Replacing Cairo required some additions, most notably in the form of an SSE-backed alpha blitter and clip regions. With these added, the compositor and terminal made for a reasonable target of the initial libc, which eventually grew to support the rest of the userspace.
With Cairo and Newlib out of the way, replacing Freetype was next on the list. The early ports to the NIH libc relied on bitmap rendering, a relic of the first builds of the OS from 2011. After much exploration and research on the inner workings of TrueType, I decided to take a different route and implemented a "signed distance field" text rasterizer. The results have thus far been surprisingly reasonable, though the renderer is not without its faults.
With the core C applications now running, the next challenge was to replace the Python userspace applications. While the Python implementations of the Panel and other apps provided a lot of functionality that still has yet to resurface in their new C counterparts, they were always intended as prototypes - always destined to be replaced sooner or later. The most challenging thing to port from Python was the menu library, which is now used throughout the OS including in the decoration library where it provides context menus, and the terminal emulator (as reflected in the visual design of this page).
A fully in-house userspace and kernel meant one final checkbox to get the OS to the "completely from scratch" stage (ending years of being merely "mostly from scratch"), and so as the last goal of the NIH project I set out to build a reasonably usable bootloader. Foregoing the completeness of a widely deployable BIOS loader, ToaruOS's new bootloader provides both an EFI payload, which should work across many machines, and a restricted BIOS loader specifically meant for live CDs running in common virtual machines.
The NIH project still needed one more thing before it could be considered complete: ports. Python, GCC, Cairo, and Freetype were must-haves for the project to be merged, and slowly the libc adapted to support them. The NIH project was concluded in October, merged with the release of ToaruOS v1.6.0, which was offered in both a "core" and "extra" distribution. Core featured only the in-house ToaruOS code, while Extra came with the legacy Python applications and Cairo and Freetype backends for the compositor and some text rendering.
New Features in 2018 #
The NIH project wasn't the only new thing that happened in 2018, as a considerable development effort went towards a plethora of new features and application improvements.
The kernel saw several improvements over the year. The most notable among them being the implementation of process groups and jobs, finally allowing us to press ^Z to suspend text editors. There were also significant improvements to POSIX file access and permissions, with cleanups and bug fixes all across the board. Driver modules also saw a lot of work, with improvements to how modules interact with procfs, and support for new filesystems (such as tar archives, which have replaced ext2 filesystem as ramdisks on live CDs).
The move from Python to C saw the reimplementation of the File Browser, which was almost completely redesigned to match the user experience found on other operating systems (particularly XFCE, which has been the primary inspiration for most of ToaruOS's visual design). Double-click interactions, custom icons for several file formats, the ability to edit files in the native text editor, and new view modes including list view and tile view, plus a navigation bar, give the new file browser a more practical and modern feel.
Beyond the kernel improvements to TTYs, the terminal emulators also saw some improvements of their own. Alternate buffers, text selection, clipboard, and menus driven by the new C menu library, provide a terminal experience that feels almost as complete as a VTE terminal under Linux. The VGA terminal wasn't left out, seeing the alternate buffer, text selection, and clipboard support as well.
One of the biggest improvements in userspace in 2018 was the work put into the text editor, Bim. Bim was fully ported to Linux (as well as macOS, FreeBSD, and Sortix) and saw almost a complete rewrite with support for syntax highlighting and dozens of new interactions. Bim is now a speedy, usable general-purpose code editor! I might even say that Bim is no longer deserving of its name (an abbreviation of "Bad IMitation (of vim)").
Underneath all of these user-visible changes, the init system also saw a rather extensive rebuild. Employing shell scripts for startup routines, new serial TTY management, and a port of the ramdisk migration system to C, the new init is a lot more like the one found in classic Unixes. The improved TTY management also allowed for the addition of a network-based remote shell, which has improved speed for headless boots considerably over the older serial approach.
Looking Towards the Future: ToaruOS in 2019 #
My goals for next year are three-fold: make ToaruOS installable, close the loop, and port everything.
Make ToaruOS installable means fixing bugs in the ext2 driver or implementing a new on-disk filesystem, as well as building tools to create and manage filesystems and partitions.
Close the loop means writing a native ToaruOS toolchain - assembler, linker, and eventually a C compiler aiming to build ToaruOS's own userspace.
Port everything means expanding our port library even farther than what was available before the NIH project - maybe even glib, or Webkit!
Patches Welcome! #
ToaruOS is Free Software, released under the terms of the NCSA / University of Illinois license. All software shipped on ToaruOS live CDs was written by the ToaruOS team, and we'd love to have you join us.
Follow ToaruOS on GitHub, and join the community on Freenode in #toaruos.
You can also follow the project on Twitter @toaruos or follow me @_klange!
Consider subscribing to my YouTube channel where I regularly post videos of new features.
The NIH Project #
For the first six years of its life, ToaruOS employed Newlib as a C standard library. When I started on ToaruOS as a university student, building everything from scratch wasn't a goal. ToaruOS made liberal use of third-party libraries in its userspace, particularly Cairo and Freetype, so the use of a third-party libc never particularly bothered me in those early years. As ToaruOS developed, though, Newlib became a detriment to progress. Adding new functionality that wasn't already present in Newlib meant adding more hacks to the Newlib headers, and further expanding an already bloated glue layer.
When I set out to replace Newlib in March, the idea of also dropping Cairo as a requirement for the compositor surfaced. ToaruOS already had a reasonably robust graphics library which was capable of providing rendering for the compositor, but it was painfully slow. Replacing Cairo required some additions, most notably in the form of an SSE-backed alpha blitter and clip regions. With these added, the compositor and terminal made for a reasonable target of the initial libc, which eventually grew to support the rest of the userspace.
With Cairo and Newlib out of the way, replacing Freetype was next on the list. The early ports to the NIH libc relied on bitmap rendering, a relic of the first builds of the OS from 2011. After much exploration and research on the inner workings of TrueType, I decided to take a different route and implemented a "signed distance field" text rasterizer. The results have thus far been surprisingly reasonable, though the renderer is not without its faults.
With the core C applications now running, the next challenge was to replace the Python userspace applications. While the Python implementations of the Panel and other apps provided a lot of functionality that still has yet to resurface in their new C counterparts, they were always intended as prototypes - always destined to be replaced sooner or later. The most challenging thing to port from Python was the menu library, which is now used throughout the OS including in the decoration library where it provides context menus, and the terminal emulator (as reflected in the visual design of this page).
A fully in-house userspace and kernel meant one final checkbox to get the OS to the "completely from scratch" stage (ending years of being merely "mostly from scratch"), and so as the last goal of the NIH project I set out to build a reasonably usable bootloader. Foregoing the completeness of a widely deployable BIOS loader, ToaruOS's new bootloader provides both an EFI payload, which should work across many machines, and a restricted BIOS loader specifically meant for live CDs running in common virtual machines.
The NIH project still needed one more thing before it could be considered complete: ports. Python, GCC, Cairo, and Freetype were must-haves for the project to be merged, and slowly the libc adapted to support them. The NIH project was concluded in October, merged with the release of ToaruOS v1.6.0, which was offered in both a "core" and "extra" distribution. Core featured only the in-house ToaruOS code, while Extra came with the legacy Python applications and Cairo and Freetype backends for the compositor and some text rendering.
New Features in 2018 #
The NIH project wasn't the only new thing that happened in 2018, as a considerable development effort went towards a plethora of new features and application improvements.
The kernel saw several improvements over the year. The most notable among them being the implementation of process groups and jobs, finally allowing us to press ^Z to suspend text editors. There were also significant improvements to POSIX file access and permissions, with cleanups and bug fixes all across the board. Driver modules also saw a lot of work, with improvements to how modules interact with procfs, and support for new filesystems (such as tar archives, which have replaced ext2 filesystem as ramdisks on live CDs).
The move from Python to C saw the reimplementation of the File Browser, which was almost completely redesigned to match the user experience found on other operating systems (particularly XFCE, which has been the primary inspiration for most of ToaruOS's visual design). Double-click interactions, custom icons for several file formats, the ability to edit files in the native text editor, and new view modes including list view and tile view, plus a navigation bar, give the new file browser a more practical and modern feel.
Beyond the kernel improvements to TTYs, the terminal emulators also saw some improvements of their own. Alternate buffers, text selection, clipboard, and menus driven by the new C menu library, provide a terminal experience that feels almost as complete as a VTE terminal under Linux. The VGA terminal wasn't left out, seeing the alternate buffer, text selection, and clipboard support as well.
One of the biggest improvements in userspace in 2018 was the work put into the text editor, Bim. Bim was fully ported to Linux (as well as macOS, FreeBSD, and Sortix) and saw almost a complete rewrite with support for syntax highlighting and dozens of new interactions. Bim is now a speedy, usable general-purpose code editor! I might even say that Bim is no longer deserving of its name (an abbreviation of "Bad IMitation (of vim)").
Underneath all of these user-visible changes, the init system also saw a rather extensive rebuild. Employing shell scripts for startup routines, new serial TTY management, and a port of the ramdisk migration system to C, the new init is a lot more like the one found in classic Unixes. The improved TTY management also allowed for the addition of a network-based remote shell, which has improved speed for headless boots considerably over the older serial approach.
Looking Towards the Future: ToaruOS in 2019 #
My goals for next year are three-fold: make ToaruOS installable, close the loop, and port everything.
Make ToaruOS installable means fixing bugs in the ext2 driver or implementing a new on-disk filesystem, as well as building tools to create and manage filesystems and partitions.
Close the loop means writing a native ToaruOS toolchain - assembler, linker, and eventually a C compiler aiming to build ToaruOS's own userspace.
Port everything means expanding our port library even farther than what was available before the NIH project - maybe even glib, or Webkit!
Patches Welcome! #
ToaruOS is Free Software, released under the terms of the NCSA / University of Illinois license. All software shipped on ToaruOS live CDs was written by the ToaruOS team, and we'd love to have you join us.
Follow ToaruOS on GitHub, and join the community on Freenode in #toaruos.
You can also follow the project on Twitter @toaruos or follow me @_klange!
Consider subscribing to my YouTube channel where I regularly post videos of new features.