Matt Chung's Site

Category: Daily Reviews

Daily Review – Day ending in 2020/09/07
Yesterday

Writing
- Blogged and published an entry describing my naive scheduler for project 1
Emotions
- I disassociated and my mind wandered into its “own world” after my wife Jess asked me to watch Elliott later that afternoon. I think part of the reason why I got so worked up had less to do with her request (cause I actually had a blast with Elliott for that 1 hour) and had to do more with an uneasy feeling caused by my being “stuck” on the school project, which translated to “I have a lot to do with no little time”
What am I grateful for?
- The fact that Jess and I can work through conflict and continue as a couple to develop ways to flesh out issues. Our system is not perfect. I like to disconnect and gather my thoughts. She likes to talk issues through and resolve them then and there. We’re meeting somewhere in the middle.
Physical and Mental Health

Elliott and daddy and boeing creek park
- Hiked with the dogs and Elliott (who sat on top of my shoulders most of the walk) and Jess at Boeing creek park, a hidden gem. That park and its thousands of 100 feet ever green trees makes you feel as if you teleported to another world, a world that reminds you how beautiful and how strong nature is. We had witnessed the powerful wind knock down one of the trees and reminded how, as humans, little we are and how we should respect nature
Music
- Came up with another beautiful sounding harmony (basically a I-IV-I-V-III-IV progression) that I should capture in my voice notes (will do that shortly)
Graduate School
- Finished up the majority of part 1 of my project, writing a scheduler in C that balances virtual CPUs on physical CPUs running on my Xen server. Only piece remaining is adding a “stabilization” feature that does not shuffle CPUs around. This should take maybe another hour of focused attention
- Read the requirements for part 2 of the project, which sounds like I’ll be emulating a “balloon” driver, a driver that expands and contracts depending on whether the system as a whole needs more or less memory, respectively
- Finished watching the lengthy video series on “CPU & Device Virtualization”
- Began watching an even lengthier video series on “Shared Memory Machines”
Organization
- Culled all the lingering, unread (and read) e-mails sleeping in my inbox. Amazing how many e-mails that I do not read so I unsubscribed from quite a few newsletters and disabled e-mail forwarding on my undergraduate academic address since I receive nothing but warning notifications around tornado weather
Work
- Nothing. At all. Thanks to the three day weekend (appreciate it labor day)
Friends and Family
- Finally remembered (thanks to revising my to do list for the day) to video call my Grandma, who hasn’t seen Elliott for a couple months now.
- Watched Elliott for an hour while Jess ran off to her appointment. Most memorable moment as a father was seeing Elliott stand up (she’s getting so good at holding this position) and with a little assistance (her gently resting her palm in mine) took one of her first steps. Who would’ve known something so little sparks so much joy
- Packed and taped up our first moving box, the box filled with all sorts of textbooks that were previously sitting on our bookshelf. Oh yea, also off the kitchen table and carried it outside, placing the table in the front yard so that anybody can pick it up (for free). These little moving victories helps me feel as if we are gaining traction and momentum with this major house move
Miscellaneous
- Talked on the phone for 45 minutes with our Loan Broker, who did an excellent job describing the different roles and parties involved in a home purchase transaction. So many terms I was unfamiliar with including: the broker, the lender, the underwriter, the title company, the escrow company. More on this on a separate blog post
Word of the day

reprisal actions on the part of his troops were brought to his notice in reports made to him

Definition

Reprisal – noun – an act of retaliation.

Today

Writing
- Publish post on notes from graduate school and/or a post on how my C skills have sharpened thanks to self studying, graduate school, switching teams at Amazon (2 years ago)
Organization
- Review OmniFocus’s forecast tab to get a sense of what meetings I have this week and any items that are due soon
Graduate School
- Continue watching lecture on “Shared memory”
Mental and Physical health
- Run for 5-10 minutes (not much but these little minutes add up) since I’ve been feeling sluggish lately and tired of seeing my belly begin to sag over my waist (physical activities are just unnecessarily harder when you pack on a few pounds)
Work
- Send e-mail to three senior engineers who will be participating in my panel
- Send meeting invite with Rahul so that they can share their presentation with me, the one that will take place in the first 10 minutes of my panel discussion
Family
- Check in with Jess since she’s right: I review my days and plan out the next ones so meticulously but fail to include her (and others) as part of my daily equation. My argument was that she’s so important and implicitly included as part of my day. But is that true? Not sure if I buy my own excuse so I’ll sit and muse on the idea.
- Review my list of “Contacts” since I haven’t touched base with many people — friends or family — since COVID-19 hit. Actually, COVID-19 has nothing to do with the fact that I rarely reach out to people. I’m so insular and in my “own world” and it’s easy to forget to reach out to loved ones
September 8, 2020
Daily Review – Day ending in 2020/09/04
Yesterday

Writing
- Blogged and published an entry for my daily review
- Blogged and published an entry explaining the libvrt’s bit map data structures that map virtual CPUs to physical CPUs
Music
- Noodled around on the guitar, playing chords in a minor key up the neck (need to start structuring and mixing up my practice sessions since they are limited to about 10-15 minutes a day — oh how fatherhood changes how we spend and maximize our alone time)
Graduate School
- Hit pause on writing C code and shifted my focus to diagramming the bitmap data structure
- Finished watching the videos lectures on “memory virtualization” and began watching the next series on CPU & Memory Virtualization
- Learned the underlying implementation of techniques such as driver ballooning and oblivious page sharing, technologies that I used to apply as a systems administrator — close to 10 years ago — when running a VMWare ESXi cluster. So awesome to be peeling off layers of the onion and gaining a deeper understanding of computer science
Work
- Met with the senior network principle engineer to discuss my design document. He agreed that the open questions I had, around failure scenarios, needed to be investigated deeper. All the points he made on data supported approach and scalability and extensibility resonated with me.
- Ordered a new develop EC2 instance, the new instance costing $110 per month (in comparison to my $25.00 instance). The portal in which I launched the instance warned me that the cost exceeds 75% of most other machines. The way I justify the decision is that I sometimes spend more time in a day watching GCC compile my code, slowing down my development velocity (which costs more than $110.00 per month).
Friends and Family
- Miss: Forgot to call my grandma using Facebook Messenger. It’s been several months since she last seen Elliott and she called me the last couple days to see her. Shucks. I even added the item to my Inbox on OmniFocus; I just didn’t reviewed my list. Oh well. Will need to come up some sort of mechanism to periodically review my calendar and my tasks throughout the day.
- Bathed Elliott last night, despite being completely shattered from waking up at 04:00 (the tiny window of focused work, whenever everyone is sleeping, even the dogs). During our bathes, we started playing this new game: I stretch out my fingers and point at my wedding ring and she tries to slide the ring off and on my finger. This game is an excellent way for her to build her motor skills and she’s improved significantly over the last couple days. It’s really cute to see her eye brows burrow when she struggles with some task and I just always try to encourage her to keep trying (and modify the game slightly to set her up for success).
Photo of the day

Mushroom looking damn beautiful

Today

As mentioned in my previous posts, I’m moving from North Seattle to Renton (the “burbs”). The entire process moving fast: in less than 3 weeks, we’ll be (fingers crossed) move into our new home that offers more space for Elliott to crawl around. We’re just waiting for the inspection report to land in our inboxes (sometime this morning) so that we can negotiate what repairs (such as replacing the water heater pressure tank) need to happen before we move in.

Organization
- Process e-mail inbox down to zero
Graduate School
- Wrap up watching lecture on CPU and device scheduling
- Run the CPU scheduling monitoring code (for Project 1) to get a sense of how the hypervisor’s default scheduler balances work out among the virtual CPUs. I’m guessing that the default behavior of the hypervisor does a decent job of scheduling work so I wonder how much my own scheduler implementation will improve utilization of the underlying physical CPUs
Mental and Physical health
- Attend acupuncture session. This will be the first acupuncture session since COVID hit back in February, almost 8 months ago. Our acupuncture closed shop, rightly slow, and only now is starting to open up her clinic.
Work
- Revisit the open comments from design review and follow up with AWS Networking teams
Family
- Check work calendar and check if I can perform the home inspection at 2:00 PM on Thursday
What are you grateful for?

Allergy season ending (no more injecting Flownase every day) and my dogs resting by my feet, providing warm company, as work remotely from home.
September 5, 2020
Daily Review – Day ending in 2020/09/03
Writing
- Blogged and published an entry for my daily review
- Blogged and published an entry on L3 Microkernel
Mental and Physical Health
- At the park, ran around in circles while holding Elliott in my arms … that sort of counts as exercise, right?
Music
- Nothing at all
Graduate School
- Migrated from my Virtualbox environment to Digital Ocean. My local Ubuntu instance kept crashing, the KVM service unable to start up the guest operating systems. The instances would just hang for no reason and instead of wasting time troubleshooting, I’d rather focus on the code itself
- Wrestled with libvrt’s documentation and finally was able to collect CPU statistics using the API (see snippet below). The documentation sometimes says you need to pass in a struct when they mean pointer (and vice versa)
```
cpuTime: 119900000000
root@aos-kvm-ubuntu-18:/tmp# ./vcpu-scheduler 
Num domains: 2
Active Domain IDs:
2
1
State: 1, nrVirtCpu: 2, cpuTime=311380000000
Domain CPU Stats:
cpuTime: 162140000000
cpuTime: 144030000000
State: 1, nrVirtCpu: 2, cpuTime=268990000000
Domain CPU Stats:
cpuTime: 142960000000
cpuTime: 119900000000
```
Work
- Firmed up my design document at work, addressing other engineer’s feedback that was sprinkled in the Quip document
- E-mailed Asians@ so that they include my upcoming event in the newsletter
Organization
- Using my ScanSnap IX1500, I scanned all my loose documents (e.g. tax statements, W-2 copies, etc) and then shredded them, clearing up space on my desk and just overall getting a little more organization
Friends and Family

Elliott getting stuck in 3 tier shelf
- Popped into Wells Fargo Bank to transfer earnest money to the escrow company. I had originally tried calling the bank at around 04:00 AM (when their support center opened) to have both enable wire transfers on my account and bump up the transfer limit but quickly realized that sending tens of thousands of dollars should be probably be done in person at the branch
- Unexpectedly watched Elliott for the about 3 hours yesterday. Jess was already running behind for the inspection for our new home located in Renton and Elliott was still napping. So instead of waking her up, she left her with me in the middle of the work day and now I truly understand the plight of parents all around the world, parents who need to work from home while taking care of young children: it’s impossible.
- Scheduled follow up veterinarian appointments for both Metric and Mushroom. Metric’s ear developed a little fissure in her ear, looking as if a tick made a home and then evacuated. And on the back of mushroom’s neck, there’s a maybe 5 or 6 white head looking bumps that are located in the place where I had applied her flea medication a couple weeks ago
Today

Organization
- Call Well’s Fargo (again) and see if they can bump the limit on my wire transfer since I want to avoid going to a Wells Fargo branch due to COVID-19
- Migrate straggling sticky notes that are sitting on desk and process them into “Writing Ideas” or “Inbox” in Omnifocus
Graduate School
- Finish memory virtualization series
- Write a few more lines of C code to get a better sense of how I’m going to write the scheduler (all a bit fuzzy right now). Not sure how scheduler is going to integrate with KVM and not sure what algorithms I’ll select and implement
Work
- Meet with principle network engineer to close out open questions on my design document
- Attend weekly operations meeting and discuss events that popped up throughout the week
Family
- Carry out morning routine: walk with Elliott and Jess and the dogs, blend up a delicious strawberry and blueberry and banana smoothie (thanks to Jess, who picked up the necessary ingredients from Trader Joe’s yesterday), feed the dogs their raw food
- Reply to loan broker and see if they can lock in the 2.875% rate for our mortgage
- Hold a discussion with Shuk (our realtor) and finalize what issues that popped up during home inspection need to get fixed before we move forward into the next step
What are you grateful for?
Spending 1 on 1 time with little Elliott yesterday (despite needing to watch her last minute during a work day). I cannot really explain it but I think I’m developing some sort of paternal love for my daughter, feelings that I’ve never experienced before, not for any human and not for any of my beloved pets. I cannot really describe the emotion … but she’s able to put a wrinkle in my nose just by sitting there and shooting me a smile.
Feelings
After 3 hours of watching Elliott yesterday, I was wiped out, taxed physically and emotionally. That little 11 month year old crawls faster than a rattlesnake and I’m having a hard time keeping up with her. She puts everything in her mouth, from dog hair to shreds of paper (but I would like to call out a victory from yesterday, a proud dad moment, when I was able to free a piece of paper from her mouth before she swallowed it: dad goals).
September 4, 2020
Daily Review – Day ending in 2020/09/02
Writing
- Blogged and published an entry for my daily review
- Blogged and published an entry that shows the steps for building librty’s documentation from source code (since the manual no longer exists on the official website)
Mental and Physical Health
- Met with my therapist, who I see every week (except last week since I had to cancel the session due to being on call)
- Shared how I enjoy mentoring other people not just on technology but on the human element of our work. That’s the good stuff.
- He thinks that my preference for having an organized mind stems from survival techniques that I developed as a young child
Music
- Practiced singing major and minor scale from memory (no instrument leading me with any tones), a skill I picked up from my guitar instructor
Graduate School
- Generated the libvrt documentation from source code
- Installed libvrt-dev, enabling me to compile the first executable binary with -lvirt flags passed to gcc
- Compiled program using example source code from documentation that I built
- Watched second part of “memory virtualization” lecture and learned that shadow pages map pages from the guest virtual operating system to machine page number
Work
- Met with someone AWS Networking , the two of us chatting about the new feature that we are going to launch in Q1 2021
- Debugged a crash with the fuzzer (frustrating because I’m unable to reproduce the crash and neither can the Principle engineer on the team)
Friends and Family
- Bathed Elliott last night, the bath running longer than usual (30 minutes instead of 10)
- Feeling very pressured and nervous around moving to Renton, so much needed to get done to make it happen (e.g. pack all of our belongings, hire a moving company, cleaning up existing house that we rent). Need to continue taking deep breathes and chip away at each task, slowly, one by one
Today

Organization
- Call Well’s Fargo (again) and see if they can bump the limit on my wire transfer since I want to avoid going to a Wells Fargo branch due to COVID-19
- Migrate straggling sticky notes that are sitting on desk and process them into “Writing Ideas” or “Inbox” in Omnifocus
Graduate School
- Wrap up memory virtualization series
- Write a few more lines of C code to get a better sense of how I’m going to write the scheduler (all a bit fuzzy right now). Not sure how scheduler is going to integrate with KVM and not sure what algorithms I’ll select and implement
Work
- E-mail Asians@ so that they include my upcoming event in the newsletter
Family
- Carry out morning routine: walk with Elliott and Jess and the dogs, blend up a delicious strawberry and blueberry and banana smoothie (thanks to Jess, who picked up the necessary ingredients from Trader Joe’s yesterday), feed the dogs their raw food
- Schedule follow up veterinarian appointments for both Metric and Mushroom
What are you grateful for?

To be in a position where I can (and have been for the last 4 years) attend therapy, thanks to my insurance covering a large portion of the bill. Everyone should be able to afford health care.

Oatmeal breakfast that Jess whipped up

Also grateful for a delicious oatmeal breakfast (above) that Jess cooked for breakfast.

Feelings
- Same as yesterday: Simultaneously excited and nervous about buying and moving into a new home
September 3, 2020
Daily Review – Day ending in 2020/09/01
Yesterday

Writing
- Blogged and published an entry for my daily review
- Blogged and published an entry that snapshots my understanding before starting on project 1 of advanced operating systems: writing a scheduler and writing a memory coordinator
Music
- E-mailed the singing instructor that I’ve been seeing for the last couple years, informing her that lately I’ve been too busy and had to shift around my priorities, now that I’ve stepped into fatherhood. I sorely miss singing and felt that the activity brought a breathe of fresh air into my life. Maybe I can continue and maybe I can do one off lessons: that’s always an option.
Graduate School
- Starting working on project 1 by ensuring that I can launch the virtual machines inside of my virtual box environment. Ran into a slew of issues that I’ve document and will publish on this blog
Work
- Presented and my design document for a new feature/service that AWS will be offering in the future. I had to shake off my nervousness, a feeling I get despite how well prepared and despite how number of years I’ve practiced and polished my public speaking skills
- Starting debugging a crash discovered by our Fuzzer. I never dealt directly with the fuzzer so this is a great learning opportunity to not only fix a problem but deeper understand what the fuzzer exactly is doing
Friends and Family

Excited to design and decorate my new home office
- Bathed Elliott last night. She only lasted about 5 minutes (about 1/2 to 1/3 of the time we usually take a bathe for) since she was so sleepy, despite her clocking in a one and a half hour nap, an hour longer than her other naps. Maybe she’s going through some sort of growth spurt? Maybe she’s sleeping better because I hung up curtains in room that shield her from the setting sun?
- Video chatted with Martin, the two of us discussing software and architecture design for an authentication system he is working on. Nice that I can share my thoughts around trade offs, trade offs that I’ve picked up from both working at Amazon over the years and from graduate school. For example, talking about the trade offs of caching and caching is not free: need to tackle cache consistency and cache coherency.
- Panicked panicked panicked. The offer that we put in on the house the day before has been accepted and my wife and I are officially pending on a new house located in Renton. Although I’m nervous and scared and will miss North Seattle, I know that this relocation is the right step for our family. Elliott needs more space and seeing her crawl around the living room — over and over and over again — reaffirms my decision. Not only that, but I can finally build myself a real work from home office, one that feels warm and one that I can call my own.
Today

Organization
- Plan day and week out by reviewing OmniFocus forecast events
- Process e-mail inbox down to zero
- Migrate sticky notes (written down while walking dogs in the morning) into writing tracker and OmniFocus
Graduate School
- Begin second series of lectures for advanced operating systems, lectures on “Memory Virtualization” (exciting stuff, I think)
Work
- Revisit the open comments from design review and follow up with AWS Networking teams
Family
- Check work calendar and check if I can perform the home inspection at 2:00 PM on Thursday
What are you grateful for?

Despite the fact that we’re in the midst of a pandemic, despite that the massive layoffs in America and 10% unemployment rate, I’m fortunate enough to be in a position to have earned and saved enough money to buy a house. I feel both very blessed and also guilty at the same time. I acknowledge my hard work and perseverance but also acknowledge that I could not have done this on my own: so many people have helped me along the way in my life. I must continue to return the favor.

Feelings
- Simultaneously excited and nervous about buying and moving into a new home
September 2, 2020
Daily Review – Day ending in 2020/08/31
Yesterday

Most interesting part of yesterday was a 1 on 1 video Chime chat that my co-worker scheduled with me. What started off with me stepping through our code base (for a set of features I added to the dataplane) morphed into a lengthy discussion around mental health and burning out and the idea of feeling confident in our own abilities. They had shared with me that they were starting to feel physically ill and after paying visits to the doctor for a full body check up, they discovered their symptoms manifest from mental health issues.

I’m very familiar with these feelings and can empathize.

In turn, I shared that I prioritize mental health and well being above all else and that I attend weekly therapy sessions (for the past four years) to deal with very similar issues that they are encountering: learning how to set boundaries, learning how to believe in oneself (i.e. imposter syndrome). It’s not as if I’ve completely overcome and mastered these problems; in fact, I wrestle with these issues — every day — but I now have have tools and data points to support me.

Although I love working as a software engineer, I sometimes wonder if I would pivot into a position or line of work where I can directly help people. It feels meaningful.

Writing
- Kept up with my writing cadence, publishing two small blog posts (a daily review and summary notes for the SPIN operating system structure)
Music
- Sang and played my own original guitar song I titled “Elliott’s in the house” for little Elliott while she was bouncing up and down on the couch while dinner, her mom guiding spoon fulls of avocado and noodles in her dime sized mouth
Graduate School
- Read first paper “SPIN”, performing a first pass on the paper (good to reinforce the learning material since there’s so much information packed into the lectures)
- Finished watching lectures on “Introduction to Virtualization” (learned that with paravirtualization, less than 2% code modification is necessary to integrate nicely with the underlying hypervisor)
Work
- Represented my team at the weekly operations meeting for my organization, stepping through all the high severity issues that I encountered and how they affected our metrics
- Finished a first draft of my design document, a paper that I am presenting to my organization today
- Met with a colleague of mine and confirmed that he will be participating in my fireside chat that I am organizing on behalf of Asians@ Amazon (I’m on the professional and development committee)
Family
- Gathered all the necessary documents (e.g. 2 months of pay stubs, last 2 years of W-2, etc) to submit to our lender since we needed to send an offer in for a house by today and needed a preapproval letter.
Today

Organization
- Plan day and week out by reviewing OmniFocus forecast events
- Process e-mail inbox down to zero
- Migrate sticky notes (written down while walking dogs in the morning) into writing tracker and OmniFocus
Graduate School
- Begin second series of lectures for advanced operating systems, lectures on “Memory Virtualization” (exciting stuff, I think)
Work
- Polish one pager design document for work and present
- Review dashboard in preparation for organization wide, weekly operations meeting
Family
- Morning routine of walking dogs at local Northacres park, blending a strawberry and banana smoothie (wait: no much coconut milk so that’s out of the question), eating lunch together, bathing Elliott as part of her night time routine
- Put together loan documents so that we can get a pre-approval for house that we are putting an offer in
Word of the day

insolent – adjective (adj) – showing a rude and arrogant lack of respect.

My strict teacher does not tolerate insolent behavior in her classroom

What are you grateful for?

I’m glad I’m writing down what I’m grateful for because its so easy for me to get trapped inside my own brain, my own world, when work (on call specifically) feels like a tornado.
- Jess being an amazing mom (so patient, so communicative) and leading parenting my example
Feelings
- I had lamented and avoided writing a paper at work and realized, after setting ink on paper for just a few minutes, I actually enjoyed the task but had to overcome my own fear of “not knowing enough” or “looking stupid” in front of my colleagues
- As mentioned above in the summary, I feel good when I can help and serve others. Not in a purely intellectual way, but mentally and emotionally. This is important and perhaps will steer me in a new (or same or different) direction with my career
September 1, 2020
Advanced Operating Systems – OS Structure – Exokernel

Unlike the SPIN OS Structure, exokernel approaches extensibility by decoupling the authorization to hardware from its actual use, by using something called secure bindings. OS libraries will request secure bindings, the Exokernel returning an encrypted key that grants that particular service to specific resources.

One other way that the exokernel manages these underlying OS services is via CPU scheduling. The exokernel maintains a linear vector “time slots”, allotting each OS service a certain amount of time to run on the CPU.

Exokernel also introduces the concept of a STLB (software translation lookaside buffer), the STLB improving performance since, during each process context switch, the exokernel will copy the hardware TLB to a software TLB structure and when the process runs again, the exokernel will copy the software TLB back into the hardware, eliminating the need for a TLB flush.

Exokernel Approach to Extensibility

Summary

Library OS requests access to a specific hardware resource. If access granted, exokernel returns an encrypted key which will be used for future keys

Examples of Candidate Resources

Summary

An OS library will perform some action, sending the Exokernel its encrypted key, and once Exokernel accepts that request, the data processing is cheap. Like packet processing: OS library requests to install predicates for packet filtering; once predicates installed, exokernel will invoke those rules on behalf.

Implementing Secure Bindings

Summary

There are three mechanisms for implementing secure bindings. First is hardware mechanisms, like fetching the TLB. Second would be software, like caching the hardware TLB inside of the OS, avoiding the cost of flushing the TLB during a context switch. And third would be downloading code into the kernel, a feature analogous to SPIN’s approach with logical domains. I don’t really understand these concepts quite yet: it’s all a bit abstract, for now.

Default Core Services in Exokernel

Default Core Services in Exokernel

Summary

For memory management, the library OS will handle the page fault (uncalled through the register) by presenting a mapping (with the secure binding) to the Exokernel, the Exokernel installing the mapping in the TLB (hardware), this step requiring privileged access.

Secure Binding

Summary

As the instructor put it, this is all a bit dicey. Both SPIN and Exokernel allow library OS privilege for pure performance but how do we guarantee that the insertion of code into the kernel is done … securely? Safely? Will find out soon, probably over the next couple videos

Memory Management using Software TLB

Software TLB

Summary

In exokernel, during a context switch, the hardware TLB for a process is copied into a STLB (software TLB) and during a context switch, new process’s software TLB will be preloaded by the Exokernel. Of course, if there’s a TLB cache miss, then the standard page fault workflow happens

Default Core Services in Exokernel (continued)

Default Core Services continued

Summary

Three is a linear vector of “time slots”, each OS service reserving a slot for its service. If the OS service runs longer than the allotted quantum, the exokernel will punish it, reducing its time slot in the next run. Time is bounded to perform the saving of the context.

Revocation of Resources

Summary

Exokernel can revoke or reclaim permission from a Library OS, the OS sending an up call (and passing a repossession vector) to the Library OS, informing the library OS that it may need to stash away its resources (on disk).

Putting it all together

Summary

While a library OS’s thread is running, an external event will be kicked up to the running process to determine what to do next

Exokernel Data Structures

Exokernel Data Structures

Summary

Exokernel maintains a data structure for each of the running library OS, maintaining a software TLB that gets loaded during a context switch

Performance Results of Spin and Exokernel

Summary

Absolute numbers are meaningless: it’s the trends that count.

September 1, 2020
Daily Review – Day ending in 2020/08/30
Yesterday

Writing
- Wrote and published an article on Time Management Tip #1: Pomodoro Technique. How I use the technique and how I’ve adjusted it to fit my needs.
Music
- Had my bi-weekly virtual guitar lesson with Jared. This was the first lesson back, after he took a 1.5 month sabbatical, a vacation he deserves and a vacation that he takes once a year. During the lesson, we didn’t actually focus on any mechanics of guitar this time, which was fine by me because instead, we had a lengthy discussion around where we want to take guitar lessons: what’s the point and what do I want to work towards. More on this in a separate blog post.
Graduate School
- Submitted both homework assignments (apparently a day before it was due: so hooray me) and watched the video lectures while sitting in the back seat of the car, while my wife drove us from Seattle to Renton (see house hunting below, under family section).
Organization
- Miss: I had a scheduled call at 06:00 PM PST with a fellow class mate and I missed the chat. For two reasons: my phone died on the way home and two the event was not scheduled in my calendar, no alerts or notifications. Fortunately, he was still online at the time so we were able to meet over Google Hangouts despite me being 30 minutes late or so.
Family
- House hunted in Renton, viewing three homes and one of which we will submit an offer. We’ll go in at listing price and per our realtor, a friend of my sisters, escalate to about 20k above if there are competing offers. I could really see Jess and I living in the home, the square footage practically doubling the size of our current rental. Funny how having kids changes things. Up until a week or two ago, I opposed the idea of moving to the suburbs, opposed to moving away from Maple Leaf/North Seattle: all favorite local cafes are within walking distance, the amazing off leash dog parks like Magnuson, top notch vegan restaurants. But since COVID hit, my view has changed. Based off of the amount of time we spend at home, which is about 23 out of the 24 hours in the day, I’d love for Elliott to have a little more space, more than the 15 square feet of crawling space that we have now.
Today

Organization
- Plan day and week out by reviewing OmniFocus forecast events
- Process e-mail inbox down to zero
- Migrate sticky notes (written down while walking dogs in the morning) into writing tracker and OmniFocus
Graduate School
- Begin on lecture series for the week, the lecture focusing on virtualization (curious what this means)
- Write up notes taken over the weekend on L3 Microkernel, SPIN, and Mach, three different underlying OS structures, each with their own trade offs
Work
- Finish one pager design document (due tomorrow)
- Review dashboard in preparation for organization wide, weekly operations meeting
Family
- Morning routine of walking dogs at local Northacres park, blending a strawberry and banana smoothie (wait: no much coconut milk so that’s out of the question), eating lunch together, bathing Elliott as part of her night time routine
- Put together loan documents so that we can get a pre-approval for house that we are putting an offer in
Bonus Image

Elliott standing up at the beach

Word of the day

decry – adjective (adj) – publicly denounce

“SPIN and Exokernel used Mach as a basis for decrying Microkernel-based design”

What are you grateful for?

I’m glad I’m writing down what I’m grateful for because its so easy for me to get trapped inside my own brain, my own world, when work (on call specifically) feels like a tornado.
- To be in a financial position to even be looking to buy a home in the midst of COVID-19
Feelings
- Nervous about the move. Just like Jess. Leaving behind North Seattle is scary: we’ve grown to love the city. Will we regret it? Maybe. But the way I see it is that this decision (of moving to Renton) is not a one-way door: we can always move back and rent in North Seattle. Sure, we might not get the same awesome rental that we been living in for 4+ years but that’s a fair trade off. We are doing the right thing by listening to our gut and being heedful to our fears.
August 31, 2020
OS Structure Overview

In the context of an operating system, what does structure even mean and why is it important? Structure determines how the operating system serves application in regards to the underlying hardware and how it balances the following qualities: protection, performance, flexibility, scalability, agility, and responsiveness.

To obtain the above qualities, several designs exist including monolithic and microkernel. In a monolithic OS, there’s a division between applications running in user space vs the operating system’s core services (e.g. file systems, memory, scheduling). Taking this division a step further is microkernel design, separating the core services from the kernel, the core services all running in user space. There are trade offs between each approach. Monolithic operating systems improve performance, reducing the number of system calls between user and privileged mode, a trade off that microkernel operating systems sacrifice. However, microkernel operating systems offer greater extensibilty, allowoing applications to select which OS core services cater to their needs.

Introduction

Summary

Will study structure of OS, whatever that means. To this end, we’ll learn about SPIN and Exokernel approaches, two terms I’m unfamiliar with, and then a micro kernel)

Quiz: OS System Services

Summary

I got most of the services, apart from inter process communication (IPC) and thread scheduling, I named the others including access to I/O devices, access to network, memory management (protection, sharing, demand paging)

OS Structure

Summary

Structure determines how OS serves applications and the underlying hardware

Goals of OS Structure

Summary

There are six goals of an OS: protection (users themselves and between users, and of course the OS), performance (the time taken for OS to carry out its service), Flexibility (not one size fits all and should be able to adapt to different applications), Scalability (essentially if hardware improves, so should the speed of the operating system), Agility (how quickly can the OS respond to changing environment), responsiveness (does OS react to events, liken I/O, quickly)

Commercial OS

Summary

Apparently I’ll learn that somehow, system designers can have their cake and eat it too. That is, they do can achieve both performance without really sacrificing protection and flexibility. How? Who knows, though I should find out soon

Monolithic Structure

Summary

Monolithic OS – division between applications and OS core services (e.g. file system, memory addressing, scheduling)

Why do we call it a monolithic structure? Because the big blob (see photo) contains all the services (e.g. file system, network access, memory management, scheduling) offered by the operating system

DOSlike Structure

Summary

At a glance, the DOS like structure seems indistinguishable from monolithic. But how is it different?
Quiz DOSlike Structure Pros and Cons

Summary

Seems like DOS allows applications to run in the same space as the OS, so no trapping is necessary: just a procedure call. Trade off here is lack of protection; the OS can get corrupted by a running application.

DOSlike Structure (continued)

Summary

DOSlike structure and its lack of protection is unacceptable for general purpose OS. On the other hand, monolitic structure reduces performance, while offering protection. But we need to also consider how we can customize and tailor needs of different applications — via customization

Opportunities for customization

Summary

Learned a little more about why customization might be important. I thought that page fault handling should be the same for every process but there’s a missed opportunity here. Since the OS sort of bets on processes not requiring the entire address space, this bet might be expensive if the page fault handler (along with the eviction algorithm) runs unnecessarily. SO perhaps OS can learn more about the process and inform the cache replacement policy

Microkernel based OS Structure

Summary

Microkernel OS – OS services run in user mode too, indistinguishable from user space apps.

Why the hell do we want to overcomplicate the OS design by introducing a microkernel based OS structure? What’s the gain here? Extensibility! Previously, application 1 and application 2 would access the file system or the memory manager or the scheduler. There’s lack of extensibility here. Also, each of those servers (e.g. file system, memory management, cpu scheduler) also run in user space and will still require some sort of IPC to talk to one another, requiring the servers to talk to the micro kernel via system calls. I’ll soon find out but I think this is expensive; that is, we are sacrificing performance, I think. Will find out shortly in next lesson

Downside to the Mikrokernel

Summary

Biggest down side with microkernel approach is potential performance hit. For application to talk to file system (or other servers), a lot of IPC is needed. Data flowing between services require IPC calls in order to Marshall data in and out between applications and servers, and between servers as well

Why performance loss

Summary

Memory locality changes since we need to context switch, dramatically impeding performance. Same goes with copying data between user space and system space. This strategy is unlike monolithic, which the servers can share data easily with one another, although a little more inflexible.

What do we want?

Summary

Trade off between performance, extensibility and protection

Is it possible to get the OS to exhibit all three desirable characteristics: performance, extensibility, and protection — at the same time? Apparently, there is, but I’ll have to wait until the next module

August 27, 2020
Advanced Operating System Notes (File Systems)
Although I’ve covered file systems in previous courses, including graduate introduction of operating systems, I really enjoy learning more about this concept in a little depth. The file system’s module is packed with great material, the material introducing a high level introduction of file system and then jumps into the deep end unveils what an operating system does behind the scenes.

Main Take Aways
- The high level purpose of a file system provides three key abstractions: file, filename and directory tree.
- A developer can interface with the file system in two ways: a position based (i.e. cursor) and a memory based (i.e. block)
- C function strol converts a string to a long
- Mmap system calls maps a file on the file system to a contiguous block of memory (the second method a developer can interface with a file systeme)
- FAT (file allocation table) glues and chains disk blocks together. It is essentially a linked list (persisted as a busy bit vector table) that is great for sequential reads (i.e. traversing the linked list) but awful for random access (i.e. to get to the middle, need to traverse from head)
- EXT linux file system is based on inodes and improves random access using 12 direct pointers (13th pointer provides first level of indirection, 14th pointer second level of indirection, 15th pointer third level of direction)
- Learned about the buffer cache (i.e. write-through) and how we a journal can help stabilize the system while maintaining decent performance
Linked list and busy bit table

(more…)
August 22, 2020