If just download the libvert application development guide, click here.
How to build the documentation
libvrt broken documentation
The libvrt developer documentation link is broken (i.e. HTTP 404). But I need the development guide for my advanced OS course so I downloaded the repository and built the documentation from source. If you want to do the same (instead of downloading the PDF I rendered above) you can execute the following instructions:
YouTube’s recommendation engine suggested that I watch a video called “The Cult of Dan Lok”. Mind you, I never even heard of Dan Lok but my intuition lead me to believe that he runs some sort of pyramid scheme. Surprise surprise: he does.
Anyways, in the video below, a student of Dan Lok describes how he dumped $26,000 into an “exclusive” program and how in that program, at every step of the way, Dan Lok (or people working directly for him) upselled a new program, a new promise from rags to riches.
I seriously don’t understand why and how people fall for this sort of crap. Don’t people understand that there’s no quick and easy fix for life? And anybody who is selling you that promise is probably full of shit?
I get livid and upset that people — like Dan Lok — can take advantage of people all over the world. Granted, I understand that these victims are consenting adults but come on.
E-mailed the singing instructor that I’ve been seeing for the last couple years, informing her that lately I’ve been too busy and had to shift around my priorities, now that I’ve stepped into fatherhood. I sorely miss singing and felt that the activity brought a breathe of fresh air into my life. Maybe I can continue and maybe I can do one off lessons: that’s always an option.
Graduate School
Starting working on project 1 by ensuring that I can launch the virtual machines inside of my virtual box environment. Ran into a slew of issues that I’ve document and will publish on this blog
Work
Presented and my design document for a new feature/service that AWS will be offering in the future. I had to shake off my nervousness, a feeling I get despite how well prepared and despite how number of years I’ve practiced and polished my public speaking skills
Starting debugging a crash discovered by our Fuzzer. I never dealt directly with the fuzzer so this is a great learning opportunity to not only fix a problem but deeper understand what the fuzzer exactly is doing
Friends and Family
Excited to design and decorate my new home office
Bathed Elliott last night. She only lasted about 5 minutes (about 1/2 to 1/3 of the time we usually take a bathe for) since she was so sleepy, despite her clocking in a one and a half hour nap, an hour longer than her other naps. Maybe she’s going through some sort of growth spurt? Maybe she’s sleeping better because I hung up curtains in room that shield her from the setting sun?
Video chatted with Martin, the two of us discussing software and architecture design for an authentication system he is working on. Nice that I can share my thoughts around trade offs, trade offs that I’ve picked up from both working at Amazon over the years and from graduate school. For example, talking about the trade offs of caching and caching is not free: need to tackle cache consistency and cache coherency.
Panicked panicked panicked. The offer that we put in on the house the day before has been accepted and my wife and I are officially pending on a new house located in Renton. Although I’m nervous and scared and will miss North Seattle, I know that this relocation is the right step for our family. Elliott needs more space and seeing her crawl around the living room — over and over and over again — reaffirms my decision. Not only that, but I can finally build myself a real work from home office, one that feels warm and one that I can call my own.
Today
Organization
Plan day and week out by reviewing OmniFocus forecast events
Process e-mail inbox down to zero
Migrate sticky notes (written down while walking dogs in the morning) into writing tracker and OmniFocus
Graduate School
Begin second series of lectures for advanced operating systems, lectures on “Memory Virtualization” (exciting stuff, I think)
Work
Revisit the open comments from design review and follow up with AWS Networking teams
Family
Check work calendar and check if I can perform the home inspection at 2:00 PM on Thursday
What are you grateful for?
Despite the fact that we’re in the midst of a pandemic, despite that the massive layoffs in America and 10% unemployment rate, I’m fortunate enough to be in a position to have earned and saved enough money to buy a house. I feel both very blessed and also guilty at the same time. I acknowledge my hard work and perseverance but also acknowledge that I could not have done this on my own: so many people have helped me along the way in my life. I must continue to return the favor.
Feelings
Simultaneously excited and nervous about buying and moving into a new home
If you are executing uvt-simplestreams-libvrt you’ll need to execute the command with sudo and exercise patience (i.e. be okay with waiting 3 minutes while program runs without outputting any informational message to the standard output)
No logging to standard output/error
I had to exercise some patience when executing the command uvt-simplestreams-libvrt sync, the command that downloads OS images to the box. Basically, the command takes several minutes to complete and does not any information messages while running, leaving you wondering if any forward progress is being made.
Tip #1 – Run with sudo
If you do not run the command with sudo, the program will download images but then fail to wrtiting to the socket.
TIp #2 – Verify images have be downloaded
Once you downloaded the images, you can list all the images by using the query subcommand.
Project 1 was released last evening at 08:59 PM PST and this morning, I decided to start on the project by reading through the overview and get the lay of the land. For this project, we’ll need to deliver to operating system components: a scheduler and a memory coordinator (not even sure what that means exactly).
So what I’m doing as part of this post is just taking a snapshot of the questions I have and topics I do not understand, topics that I’ll probably understand in much more depth as the project progresses. More often than not, I often dismissive of all the work I put in over the semester and this post is one way to honor the time and commitment.
Overall, this project’s difficulty sits in the right place — not too hard but not too easy. The sweet spot for Deliberate Practice.
Questions I have
What algorithm should I implement for my scheduler?
What algorithms fit the needs for this project
What the heck is a memory coordinator?
Why do we have a memory coordinator? What’s it purpose?
How do you measure the success of a memory coordinator?
How do I use libvrt library?
What is QEMU?
Where does the scheduler sit in relationship to the operating system?
How will I get the hypervisor to invoke my scheduler versus another scheduler?
Project Requirements
You need to implement two separate C programs, one for vCPU scheduler (vcpu_scheduler.c) and another for memory coordinator (memory_coordinator.c)
Most interesting part of yesterday was a 1 on 1 video Chime chat that my co-worker scheduled with me. What started off with me stepping through our code base (for a set of features I added to the dataplane) morphed into a lengthy discussion around mental health and burning out and the idea of feeling confident in our own abilities. They had shared with me that they were starting to feel physically ill and after paying visits to the doctor for a full body check up, they discovered their symptoms manifest from mental health issues.
I’m very familiar with these feelings and can empathize.
In turn, I shared that I prioritize mental health and well being above all else and that I attend weekly therapy sessions (for the past four years) to deal with very similar issues that they are encountering: learning how to set boundaries, learning how to believe in oneself (i.e. imposter syndrome). It’s not as if I’ve completely overcome and mastered these problems; in fact, I wrestle with these issues — every day — but I now have have tools and data points to support me.
Although I love working as a software engineer, I sometimes wonder if I would pivot into a position or line of work where I can directly help people. It feels meaningful.
Writing
Kept up with my writing cadence, publishing two small blog posts (a daily review and summary notes for the SPIN operating system structure)
Music
Sang and played my own original guitar song I titled “Elliott’s in the house” for little Elliott while she was bouncing up and down on the couch while dinner, her mom guiding spoon fulls of avocado and noodles in her dime sized mouth
Graduate School
Read first paper “SPIN”, performing a first pass on the paper (good to reinforce the learning material since there’s so much information packed into the lectures)
Finished watching lectures on “Introduction to Virtualization” (learned that with paravirtualization, less than 2% code modification is necessary to integrate nicely with the underlying hypervisor)
Work
Represented my team at the weekly operations meeting for my organization, stepping through all the high severity issues that I encountered and how they affected our metrics
Finished a first draft of my design document, a paper that I am presenting to my organization today
Met with a colleague of mine and confirmed that he will be participating in my fireside chat that I am organizing on behalf of Asians@ Amazon (I’m on the professional and development committee)
Family
Gathered all the necessary documents (e.g. 2 months of pay stubs, last 2 years of W-2, etc) to submit to our lender since we needed to send an offer in for a house by today and needed a preapproval letter.
Today
Organization
Plan day and week out by reviewing OmniFocus forecast events
Process e-mail inbox down to zero
Migrate sticky notes (written down while walking dogs in the morning) into writing tracker and OmniFocus
Graduate School
Begin second series of lectures for advanced operating systems, lectures on “Memory Virtualization” (exciting stuff, I think)
Work
Polish one pager design document for work and present
Review dashboard in preparation for organization wide, weekly operations meeting
Family
Morning routine of walking dogs at local Northacres park, blending a strawberry and banana smoothie (wait: no much coconut milk so that’s out of the question), eating lunch together, bathing Elliott as part of her night time routine
Put together loan documents so that we can get a pre-approval for house that we are putting an offer in
Word of the day
insolent – adjective (adj) – showing a rude and arrogant lack of respect.
My strict teacher does not tolerate insolent behavior in her classroom
What are you grateful for?
I’m glad I’m writing down what I’m grateful for because its so easy for me to get trapped inside my own brain, my own world, when work (on call specifically) feels like a tornado.
Jess being an amazing mom (so patient, so communicative) and leading parenting my example
Feelings
I had lamented and avoided writing a paper at work and realized, after setting ink on paper for just a few minutes, I actually enjoyed the task but had to overcome my own fear of “not knowing enough” or “looking stupid” in front of my colleagues
As mentioned above in the summary, I feel good when I can help and serve others. Not in a purely intellectual way, but mentally and emotionally. This is important and perhaps will steer me in a new (or same or different) direction with my career
Unlike the SPIN OS Structure, exokernel approaches extensibility by decoupling the authorization to hardware from its actual use, by using something called secure bindings. OS libraries will request secure bindings, the Exokernel returning an encrypted key that grants that particular service to specific resources.
One other way that the exokernel manages these underlying OS services is via CPU scheduling. The exokernel maintains a linear vector “time slots”, allotting each OS service a certain amount of time to run on the CPU.
Exokernel also introduces the concept of a STLB (software translation lookaside buffer), the STLB improving performance since, during each process context switch, the exokernel will copy the hardware TLB to a software TLB structure and when the process runs again, the exokernel will copy the software TLB back into the hardware, eliminating the need for a TLB flush.
Exokernel Approach to Extensibility
Summary
Library OS requests access to a specific hardware resource. If access granted, exokernel returns an encrypted key which will be used for future keys
Examples of Candidate Resources
Summary
An OS library will perform some action, sending the Exokernel its encrypted key, and once Exokernel accepts that request, the data processing is cheap. Like packet processing: OS library requests to install predicates for packet filtering; once predicates installed, exokernel will invoke those rules on behalf.
Implementing Secure Bindings
Summary
There are three mechanisms for implementing secure bindings. First is hardware mechanisms, like fetching the TLB. Second would be software, like caching the hardware TLB inside of the OS, avoiding the cost of flushing the TLB during a context switch. And third would be downloading code into the kernel, a feature analogous to SPIN’s approach with logical domains. I don’t really understand these concepts quite yet: it’s all a bit abstract, for now.
Default Core Services in Exokernel
Default Core Services in Exokernel
Summary
For memory management, the library OS will handle the page fault (uncalled through the register) by presenting a mapping (with the secure binding) to the Exokernel, the Exokernel installing the mapping in the TLB (hardware), this step requiring privileged access.
Secure Binding
Summary
As the instructor put it, this is all a bit dicey. Both SPIN and Exokernel allow library OS privilege for pure performance but how do we guarantee that the insertion of code into the kernel is done … securely? Safely? Will find out soon, probably over the next couple videos
Memory Management using Software TLB
Software TLB
Summary
In exokernel, during a context switch, the hardware TLB for a process is copied into a STLB (software TLB) and during a context switch, new process’s software TLB will be preloaded by the Exokernel. Of course, if there’s a TLB cache miss, then the standard page fault workflow happens
Default Core Services in Exokernel (continued)
Default Core Services continued
Summary
Three is a linear vector of “time slots”, each OS service reserving a slot for its service. If the OS service runs longer than the allotted quantum, the exokernel will punish it, reducing its time slot in the next run. Time is bounded to perform the saving of the context.
Revocation of Resources
Summary
Exokernel can revoke or reclaim permission from a Library OS, the OS sending an up call (and passing a repossession vector) to the Library OS, informing the library OS that it may need to stash away its resources (on disk).
Putting it all together
Summary
While a library OS’s thread is running, an external event will be kicked up to the running process to determine what to do next
Exokernel Data Structures
Exokernel Data Structures
Summary
Exokernel maintains a data structure for each of the running library OS, maintaining a software TLB that gets loaded during a context switch
Performance Results of Spin and Exokernel
Summary
Absolute numbers are meaningless: it’s the trends that count.
The concept of border crossing pops up over and over again. This is a new term I never heard of prior to this class. The term is almost synonymous to a context switch but it is subtly different in the sense that a context switch (switch from one process to another or one thread to another) can occur without a border crossing, without changing the underlying hardware address space.
SPIN attempts to enforce protection at the compiler level, by using a restrictive language called Modula-3. Unlike the C language, where you can cast a pointer to whatever data structure you like, Modula-3 enforces type safety, only allowing the developer to cast a pointer to specific types of data structures that they had already specified earlier in the code.
SPIN offers extensibility by allowing different types of OS services to co-exist with one another.
But what are the trade offs with SPIN, when compared with Microkernel and Exokernel?
It appears that SPIN would be more performant than Microkernel due to no border crossings while maintaining flexibilty (multiple types of OS services that cater to application processes) and security (via logical protection domains) with Modula-3, allowing code OS services library code to co-locate with kernel code.
Introduction
Customizing OS with SPIN
Summary
SPIN and Exokernel take two different paths to achieving extensibility. These designs overcome the issue of Microkernel, which compromises in performance due to border crossings, and monolithic, which does not lend itself to extensibility
What we are shooting for in OS Structure
Summary
We want flexibility similar to a microkernel based approach but also want protection and performance of monolithic. We want the best of both worlds: performance protection flexibility
Approaches to Extensibility
Damn, this is a super long video (8 minutes, compared to the other one to two minute videos prior)
Capability based
Hydra OS (1981)
Summary
Hydra did not fully achieve its goal of extensibility
Micro Kernel Based
Summary
Performance took a back seat, since the focus was on extensibility and portability. Bad press for micro kernel based due to the twin goals.
SPIN approach to extensibility
Summary
By co locating the kernel and extension in the same hardware space, the extensions are cheap as procedure call. Doing this by depending on a strongly typed language to provide safety
Logical Protection Domains
Summary
Using a programing language called Modula3, which doesn’t appear to be popular in practice, we can enforce protection at the logical level. This programming language, unlike C, restricts casting pointers to random data structures, only allowing the cast to a particular data type.
Spin mechanisms for protection domains
Summary
The secret sauce of protection and performance are the mechanisms of creating (i.e. expose entry points), resolving (i.e. leverage other entry points), and combining of logical protection domains
Customized OS with Spin
Another example of SPIN os customization
Summary
There can be multiple file systems (written in Modula3), each file system catering to their callers, and each file system using the same underlying hardware address space. And they can share modules with one another, like the networking entry point.
Example Extensions
Summary
Example of Unix Servers implementing their OS on SPIN as well as a video server / display client building on top of spin
Quiz: Border Crossings
Quiz: Least likely for border crossing
Summary
Microkernel and SPIN offer performance since they limit the border crossings. In SPIN, Logical domains do not require border crossings
SPIN Mechanisms for Events
SPIN classifies three types of event handling: one-to-one, one-to-many, many-to-one
Summary
To handle events (like packet arrival) we can have a 1:1 mapping, 1:N mapping or N:1 mapping. For 1:1, an example would be an ICMP packet arriving and the 1 ICMP handler running. In a 1:N mapping, the IP packet arrived event runs and signals three other event handlers like ICMP, UDP, or TCP. Then finally, there is a N:1, and an example of this is an Ethernet and ATM packet event arrives but both funnel into the IP handler
Default Core Services in SPIN
Summary
SPIN offers core services like memory management, CPU scheduling etc. And SPIN will provide a header file that OS programmers need to implement. Remember: these implementations talk to each other through well defined interfaces, offering protection, but are also performant cause there are no border crossings)
Default Core Services in SPIN (Continued)
Summary
Provides primitives, the interface function definition. The semantics are up to the extension itself. SPIN makes sure extensions get time on scheduler
Wrote and published an article on Time Management Tip #1: Pomodoro Technique. How I use the technique and how I’ve adjusted it to fit my needs.
Music
Had my bi-weekly virtual guitar lesson with Jared. This was the first lesson back, after he took a 1.5 month sabbatical, a vacation he deserves and a vacation that he takes once a year. During the lesson, we didn’t actually focus on any mechanics of guitar this time, which was fine by me because instead, we had a lengthy discussion around where we want to take guitar lessons: what’s the point and what do I want to work towards. More on this in a separate blog post.
Graduate School
Submitted both homework assignments (apparently a day before it was due: so hooray me) and watched the video lectures while sitting in the back seat of the car, while my wife drove us from Seattle to Renton (see house hunting below, under family section).
Organization
Miss: I had a scheduled call at 06:00 PM PST with a fellow class mate and I missed the chat. For two reasons: my phone died on the way home and two the event was not scheduled in my calendar, no alerts or notifications. Fortunately, he was still online at the time so we were able to meet over Google Hangouts despite me being 30 minutes late or so.
Family
House hunted in Renton, viewing three homes and one of which we will submit an offer. We’ll go in at listing price and per our realtor, a friend of my sisters, escalate to about 20k above if there are competing offers. I could really see Jess and I living in the home, the square footage practically doubling the size of our current rental. Funny how having kids changes things. Up until a week or two ago, I opposed the idea of moving to the suburbs, opposed to moving away from Maple Leaf/North Seattle: all favorite local cafes are within walking distance, the amazing off leash dog parks like Magnuson, top notch vegan restaurants. But since COVID hit, my view has changed. Based off of the amount of time we spend at home, which is about 23 out of the 24 hours in the day, I’d love for Elliott to have a little more space, more than the 15 square feet of crawling space that we have now.
Today
Organization
Plan day and week out by reviewing OmniFocus forecast events
Process e-mail inbox down to zero
Migrate sticky notes (written down while walking dogs in the morning) into writing tracker and OmniFocus
Graduate School
Begin on lecture series for the week, the lecture focusing on virtualization (curious what this means)
Write up notes taken over the weekend on L3 Microkernel, SPIN, and Mach, three different underlying OS structures, each with their own trade offs
Work
Finish one pager design document (due tomorrow)
Review dashboard in preparation for organization wide, weekly operations meeting
Family
Morning routine of walking dogs at local Northacres park, blending a strawberry and banana smoothie (wait: no much coconut milk so that’s out of the question), eating lunch together, bathing Elliott as part of her night time routine
Put together loan documents so that we can get a pre-approval for house that we are putting an offer in
Bonus Image
Elliott standing up at the beach
Word of the day
decry – adjective (adj) – publicly denounce
“SPIN and Exokernel used Mach as a basis for decrying Microkernel-based design”
What are you grateful for?
I’m glad I’m writing down what I’m grateful for because its so easy for me to get trapped inside my own brain, my own world, when work (on call specifically) feels like a tornado.
To be in a financial position to even be looking to buy a home in the midst of COVID-19
Feelings
Nervous about the move. Just like Jess. Leaving behind North Seattle is scary: we’ve grown to love the city. Will we regret it? Maybe. But the way I see it is that this decision (of moving to Renton) is not a one-way door: we can always move back and rent in North Seattle. Sure, we might not get the same awesome rental that we been living in for 4+ years but that’s a fair trade off. We are doing the right thing by listening to our gut and being heedful to our fears.
