After several conversations with @djwillia @mato @avsm and others, turns out @ricarkol already did a prototype and actually got IncludeOS up and running on ukvm - which is awesome! We’d now like to try and get this work upstream and were discussing whether to add support for ukvm directly or to go via the Solo5 interface. I’d have thought a direct integration with ukvm was simpler, but then again Solo5 might add a layer of abstraction that could allow ukvm to change more rapidly. What are the pro’s and cons?
The main point in favor of having IncludeOS on top of Solo5 is that Solo5 would hide a lot of details. For example, the hypercall mechanism is already different between architectures: ukvm in x86 uses PIO and arm uses MMIO. The ukvm interface might also change, specially ports and the mechanism used to implement hypercalls. Another point in favor is that by coding against Solo5, IncludeOS would get muen; and ukvm for arm, freebsd vmm, and macos for free.
The issue with having IncludeOS on top of Solo5 is that both Solo5 and IncludeOS implement many of the same features: memory management, interrupt handling, startup, virtio (if Solo5 is built in virtio mode). The cleanest approach would be to use the Solo5 version of all of these, but IncludeOS have some of those enhanced for better performance (like virtio). So, we would have to use a complicated mix from both sides, or disable IncludeOS on solo5_virtio.
I’m not familiar with the IncludeOS architecture in detail, but I would treat Solo5 as a low-level “Hardware Abstraction Layer”. Also, to address @ricarkol’s point about virtio, I would ignore it as a “supported” target from the IncludeOS point of view as you already have your own virtio implementations.
We’re open to patches / requests to change the Solo5 APIs if there are major blockers for your porting efforts. Note that the current APIs are still to be considered unstable.
@mato @ricarkol sounds like Solo5 then, thanks - for ukvm it looked like the calls were pretty thin shims anyway. For virtio it’d be interesting to see if we could consolidate some things. Although it looks like Solo5 is using PIC / PIT instead of APIC / MSIx etc.? I think we mainly switched to APIC for high resolution timers, but IRQ performance should be improved as well. Any plans on that end? And oh, muen looks cool!