Author: mattchung

  • L3 Microkernel

    L3 Microkernel

    I learned that with an L3 Microkernel approach, each OS runs in their own address space and that they are indistinguishable from the end-user applications running in user land. Because they run in user land, it seems intuitive that this kill performance due to border crossings (not just necessarily context switching, but address space switching and inter process communication). Turns out that this performance loss has been debunked and that border crossings do not cost 900 cycles — this number can be dropped to around 100 cycles if the mikrokernel, unlike Mach, stray away from writing code that is platform dependent. In other words, the expensive border crossing was a result of a heavy code base, code with portability as one of its main goals.

    L02d: The L3 Microkernel Approach

    Summary

    Learned why you might not need to flush TLB on context switch with address space tag in the TLB.

    Introduction

    Summary

    Focus on this lesson is evaluating L3, a micro kernel based design, a system and design with a contrarian view point

    Microkernel-Based OS Structure

    Summary

    Each of the OS services run in their own address space, the services indistinguishable from applications running in user space

    Potentials for Performance Loss

    Potentials for performance loss

    Summary

    The primary performance killer is border crossings, between user space and privileged space, a crossing required for user applications and well as operating system services. Also, that’s the explicit cost. There’s an implicit cost of procedure calls as well: loss of locality

    L3 Microkernel

    L3 Microkernel

    Summary

    L3 Microkernel argues that the OS structure is sound but what really needs to be the focus is efficient implementation since its possible to share hardware address space while offering protection domains for the OS services. How that works? Will find out soon

    Strikes against the Microkernel

    Summary

    Three explicit strikes of providing application level service in a microkernel structure (i.e. border crossings, address space switches, thread switches and IPC requiring the kernel. One implicit cost: memory subsystem and loss of locality

    Debunking User Kernel Borer Crossing Myth

    Debunking user <-> kernel borer crossing myth
    Memory Effects

    Summary

    SPIN and Exokernel used Mach as basis for decrying micro kernel based design. In reality, the border crossing costs is much cheaper: 123 processor cycles versus 900 cycles (in March)

    Address Space Switches

    Summary

    For address space switch, do you need to flush the TLB? It depends. I learned that for address space tags (basically a flag for a process ID), you do not have to. But I also thought that if you used a virtually indexed physically tagged address you don’t have to either.

    Address Space Switches with AS Tagged TLB

    Summary

    With address space tags (discussed in previous section), you do not need to flush the TLB during a context switch. This is because the hardware will not check two fields: the tag and the AS (i.e. the process ID). If and only if those two attributes match do we have a TLB hit).

    Liedke’s Suggestions for Avoiding a TLB Flush

    Summary

    Basically leverage the hardware’s capabilities, like segment registers. By defining a protection domain, the hardware will check whether or not the virtual address being translated falls between the lower and upper bound defined in the segment registered, the base and bound registers. Still don’t understand how/why we can use multiple protection domains.

    Large Protection Domains

    Summary

    If a process occupies all the hardware, then there are explicit and implicit costs. Explicit in the sense that a TLB flush may take up to 800+ CPU cycles. Implicit in the sense that we lose locality

    Upshot for Address space switching?

    Summary

    Small address space then no problem (in terms of expensive costs) for context switching. Large address space becomes problematic because of not only flushing the TLB but more costly is the lost in locality, not having a warm cache

    Thread switches and IPC

    Summary

    The third myth that L3 Microkernel debunks is the expensive cost of thread switching. By construction (not entirely sure what this means or how this is proved)

    Memory Effects

    Summary

    Implicit costs can be debunked by putting protection domains in the same hardware address space, but requires that we use segment registers to protect processes from one another. For large protection domains, the costs cannot be avoided

    Reasons for Mach’s expensive border crossing

    Summary

    Because Mach’s focus was on portability, there’s code bloat and as a result, there’s less locality and incurs longer latency for border crossing. In short, Mach’s memory footprint (due to code bloat and portability) is the main culprit, not the actual philosophy of micro kernel

  • Daily Review – Day ending in 2020/09/02

    Daily Review – Day ending in 2020/09/02

    Writing

    Mental and Physical Health

    • Met with my therapist, who I see every week (except last week since I had to cancel the session due to being on call)
    • Shared how I enjoy mentoring other people not just on technology but on the human element of our work. That’s the good stuff.
    • He thinks that my preference for having an organized mind stems from survival techniques that I developed as a young child

    Music

    • Practiced singing major and minor scale from memory (no instrument leading me with any tones), a skill I picked up from my guitar instructor

    Graduate School

    • Generated the libvrt documentation from source code
    • Installed libvrt-dev, enabling me to compile the first executable binary with -lvirt flags passed to gcc
    • Compiled program using example source code from documentation that I built
    • Watched second part of “memory virtualization” lecture and learned that shadow pages map pages from the guest virtual operating system to machine page number

    Work

    • Met with someone AWS Networking , the two of us chatting about the new feature that we are going to launch in Q1 2021
    • Debugged a crash with the fuzzer (frustrating because I’m unable to reproduce the crash and neither can the Principle engineer on the team)

    Friends and Family

     

    • Bathed Elliott last night, the bath running longer than usual (30 minutes instead of 10)
    • Feeling very pressured and nervous around moving to Renton, so much needed to get done to make it happen (e.g. pack all of our belongings, hire a moving company, cleaning up existing house that we rent). Need to continue taking deep breathes and chip away at each task, slowly, one by one

    Today

    Organization

    • Call Well’s Fargo (again) and see if they can bump the limit on my wire transfer since I want to avoid going to a Wells Fargo branch due to COVID-19
    • Migrate straggling sticky notes that are sitting on desk and process them into “Writing Ideas” or “Inbox” in Omnifocus

    Graduate School

    • Wrap up memory virtualization series
    • Write a few more lines of C code to get a better sense of how I’m going to write the scheduler (all a bit fuzzy right now). Not sure how scheduler is going to integrate with KVM and not sure what algorithms I’ll select and implement

    Work

    • E-mail Asians@ so that they include my upcoming event in the newsletter

    Family

    • Carry out morning routine: walk with Elliott and Jess and the dogs, blend up a delicious strawberry and blueberry and banana smoothie (thanks to Jess, who picked up the necessary ingredients from Trader Joe’s yesterday), feed the dogs their raw food
    • Schedule follow up veterinarian appointments for both Metric and Mushroom

    What are you grateful for?

    To be in a position where I can (and have been for the last 4 years) attend therapy, thanks to my insurance covering a large portion of the bill. Everyone should be able to afford health care.

    Oatmeal breakfast that Jess whipped up

    Also grateful for a delicious oatmeal breakfast (above) that Jess cooked for breakfast.

    Feelings

    • Same as yesterday: Simultaneously excited and nervous about buying and moving into a new home
  • How to build the libvrt documentation from source

    How to build the libvrt documentation from source

    If just download the libvert application development guide, click here.

    How to build the documentation

    libvrt broken documentation

     

    The libvrt developer documentation link is broken (i.e. HTTP 404). But I need the development guide for my advanced OS course so I downloaded the repository and built the documentation from source. If you want to do the same (instead of downloading the PDF I rendered above) you can execute the following instructions:

    [code lang=”bash”]
    git clone https://libvirt.org/git/libvirt-appdev-guide.git
    sudo apt install publican
    publican build –langs=en-US –formats=pdf –brand_dir=./
    [/code]

     

  • Why do people fall prey to pyramid schemes?

    Why do people fall prey to pyramid schemes?

    YouTube’s recommendation engine suggested that I watch a video called “The Cult of Dan Lok”. Mind you, I never even heard of Dan Lok but my intuition lead me to believe that he runs some sort of pyramid scheme. Surprise surprise: he does.

    Anyways, in the video below, a student of Dan Lok describes how he dumped $26,000 into an “exclusive” program and how in that program, at every step of the way, Dan Lok (or people working directly for him) upselled a new program, a new promise from rags to riches.

    I seriously don’t understand why and how people fall for this sort of crap. Don’t people understand that there’s no quick and easy fix for life? And anybody who is selling you that promise is probably full of shit?

    I get livid and upset that people — like Dan Lok — can take advantage of people all over the world. Granted, I understand that these victims are consenting adults but come on.

    Anyways, when some time frees up (from graduate school and work and raising a child … so never), I’d love to dig more into this subject and maybe even read a book titled: “False Profits: Seeking Financial and Spiritual Deliverance in Multi-Level Marketing and Pyramid Schemes”

  • How to launch and ssh to a KVM instance

    How to launch and ssh to a KVM instance

    This posts assumes that you’ve already synced the OS images on to your virtual box. If you haven’t, check out this blog post.

    While launching a new virtual machine for project1, I kept on running into variants of the following error:

    vagrant@ubuntu-bionic:~$ sudo uvt-kvm ssh aos_vm1 
    ubuntu@192.168.122.183: Permission denied (publickey).

    To get this to work, here are the steps I took:

    vagrant@ubuntu-bionic:~$ sudo uvt-kvm create aos_vm1 release=bionic --memory 256 --ssh-public-key-file /root/.ssh/id_rsa.pub
    vagrant@ubuntu-bionic:~$ sudo uvt-kvm wait aos_vm1
    vagrant@ubuntu-bionic:~$ sudo uvt-kvm ssh aos_vm1
  • Daily Review – Day ending in 2020/09/01

    Daily Review – Day ending in 2020/09/01

    Yesterday

    Writing

    Music

    • E-mailed the singing instructor that I’ve been seeing for the last couple years, informing her that lately I’ve been too busy and had to shift around my priorities, now that I’ve stepped into fatherhood. I sorely miss singing and felt that the activity brought a breathe of fresh air into my life. Maybe I can continue and maybe I can do one off lessons: that’s always an option.

    Graduate School

    • Starting working on project 1 by ensuring that I can launch the virtual machines inside of my virtual box environment. Ran into a slew of issues that I’ve document and will publish on this blog

    Work

    • Presented and my design document for a new feature/service that AWS will be offering in the future. I had to shake off my nervousness, a feeling I get despite how well prepared and despite how number of years I’ve practiced and polished my public speaking skills
    • Starting debugging a crash discovered by our Fuzzer. I never dealt directly with the fuzzer so this is a great learning opportunity to not only fix a problem but deeper understand what the fuzzer exactly is doing

    Friends and Family

    Excited to design and decorate my new home office
    • Bathed Elliott last night. She only lasted about 5 minutes (about 1/2 to 1/3 of the time we usually take a bathe for) since she was so sleepy, despite her clocking in a one and a half hour nap, an hour longer than her other naps. Maybe she’s going through some sort of growth spurt? Maybe she’s sleeping better because I hung up curtains in room that shield her from the setting sun?
    • Video chatted with Martin, the two of us discussing software and architecture design for an authentication system he is working on. Nice that I can share my thoughts around trade offs, trade offs that I’ve picked up from both working at Amazon over the years and from graduate school. For example, talking about the trade offs of caching and caching is not free: need to tackle cache consistency and cache coherency.
    • Panicked panicked panicked. The offer that we put in on the house the day before has been accepted and my wife and I are officially pending on a new house located in Renton. Although I’m nervous and scared and will miss North Seattle, I know that this relocation is the right step for our family. Elliott needs more space and seeing her crawl around the living room — over and over and over again — reaffirms my decision. Not only that, but I can finally build myself a real work from home office, one that feels warm and one that I can call my own.

    Today

    Organization

    • Plan day and week out by reviewing OmniFocus forecast events
    • Process e-mail inbox down to zero
    • Migrate sticky notes (written down while walking dogs in the morning) into writing tracker and OmniFocus

    Graduate School

    • Begin second series of lectures for advanced operating systems, lectures on “Memory Virtualization” (exciting stuff, I think)

    Work

    • Revisit the open comments from design review and follow up with AWS Networking teams

    Family

    • Check work calendar and check if I can perform the home inspection at 2:00 PM on Thursday

    What are you grateful for?

    Despite the fact that we’re in the midst of a pandemic, despite that the massive layoffs in America and 10% unemployment rate,  I’m fortunate enough to be in a position to have earned and saved enough money to buy a house. I feel both very blessed and also guilty at the same time. I acknowledge my hard work and perseverance but also acknowledge that I could not have done this on my own: so many people have helped me along the way in my life. I must continue to return the favor.

    Feelings

    • Simultaneously excited and nervous about buying and moving into a new home
  • Syncing releases with uvt-simplestreams-libvrt

    Syncing releases with uvt-simplestreams-libvrt

    If you are executing uvt-simplestreams-libvrt you’ll need to execute the command with sudo and exercise patience (i.e. be okay with waiting 3 minutes while program runs without outputting any informational message to the standard output)

    No logging to standard output/error

    I had to exercise some patience when executing the command uvt-simplestreams-libvrt sync, the command that downloads OS images to the box. Basically, the command takes several minutes to complete and does not any information messages while running, leaving you wondering if any forward progress is being made.

    Tip #1 – Run with sudo

    If you do not run the command with sudo, the program will download images but then fail to wrtiting to the socket.

    TIp #2 – Verify images have be downloaded

    Once you downloaded the images, you can list all the images by using the query subcommand.

     

  • A snapshot of my understanding before beginning project 1 (scheduler, memory coordinator)

    A snapshot of my understanding before beginning project 1 (scheduler, memory coordinator)

    Project 1 was released last evening at 08:59 PM PST and this morning, I decided to start on the project by reading through the overview and get the lay of the land. For this project, we’ll need to deliver to operating system components: a scheduler and a memory coordinator (not even sure what that means exactly).

    So what I’m doing as part of this post is just taking a snapshot of the questions I have and topics I do not understand, topics that I’ll probably understand in much more depth as the project progresses. More often than not, I often dismissive of all the work I put in over the semester and this post is one way to honor the time and commitment.

    Overall, this project’s difficulty sits in the right place — not too hard but not too easy. The sweet spot for Deliberate Practice.

    Questions I have

    • What algorithm should I implement for my scheduler?
    • What algorithms fit the needs for this project
    • What the heck is a memory coordinator?
    • Why do we have a memory coordinator? What’s it purpose?
    • How do you measure the success of a memory coordinator?
    • How do I use libvrt library?
    • What is QEMU?
    • Where does the scheduler sit in relationship to the operating system?
    • How will I get the hypervisor to invoke my scheduler versus another scheduler?

    Project Requirements

    • You need to implement two separate C programs, one for vCPU scheduler (vcpu_scheduler.c) and another for memory coordinator (memory_coordinator.c)

    References

    1. Introduction to QEMU
  • Daily Review – Day ending in 2020/08/31

    Daily Review – Day ending in 2020/08/31

    Yesterday

    Most interesting part of yesterday was a 1 on 1 video Chime chat that my co-worker scheduled with me. What started off with me stepping through our code base (for a set of features I added to the dataplane) morphed into a lengthy discussion around mental health and burning out and the idea of feeling confident in our own abilities. They had shared with me that they were starting to feel physically ill and after paying visits to the doctor for a full body check up, they discovered their symptoms manifest from mental health issues.

    I’m very familiar with these feelings and can empathize.

    In turn, I shared that I prioritize mental health and well being above all else and that I attend weekly therapy sessions (for the past four years) to deal with very similar issues that they are encountering: learning how to set boundaries, learning how to believe in oneself (i.e. imposter syndrome). It’s not as if I’ve completely overcome and mastered these problems; in fact, I wrestle with these issues — every day — but I now have have tools and data points to support me.

    Although I love working as a software engineer, I sometimes wonder if I would pivot into a position or line of work where I can directly help people. It feels meaningful.

    Writing

    • Kept up with my writing cadence, publishing two small blog posts (a daily review and summary notes for the SPIN operating system structure)

    Music

    • Sang and played my own original guitar song I titled “Elliott’s in the house” for little Elliott while she was bouncing up and down on the couch while dinner, her mom guiding spoon fulls of avocado and noodles in her dime sized mouth

    Graduate School

    • Read first paper “SPIN”, performing a first pass on the paper (good to reinforce the learning material since there’s so much information packed into the lectures)
    • Finished watching lectures on “Introduction to Virtualization” (learned that with paravirtualization, less than 2% code modification is necessary to integrate nicely with the underlying hypervisor)

    Work

    • Represented my team at the weekly operations meeting for my organization, stepping through all the high severity issues that I encountered and how they affected our metrics
    • Finished a first draft of my design document, a paper that I am presenting to my organization today
    • Met with a colleague of mine and confirmed that he will be participating in my fireside chat that I am organizing on behalf of Asians@ Amazon (I’m on the professional and development committee)

    Family

    • Gathered all the necessary documents (e.g. 2 months of pay stubs, last 2 years of W-2, etc) to submit to our lender since we needed to send an offer in for a house by today and needed a preapproval letter.

    Today

    Organization

    • Plan day and week out by reviewing OmniFocus forecast events
    • Process e-mail inbox down to zero
    • Migrate sticky notes (written down while walking dogs in the morning) into writing tracker and OmniFocus

    Graduate School

    • Begin second series of lectures for advanced operating systems, lectures on “Memory Virtualization” (exciting stuff, I think)

    Work

    • Polish one pager design document for work and present
    • Review dashboard in preparation for organization wide, weekly operations meeting

    Family

    • Morning routine of walking dogs at local Northacres park, blending a strawberry and banana smoothie (wait: no much coconut milk so that’s out of the question), eating lunch together, bathing Elliott as part of her night time routine
    • Put together loan documents so that we can get a pre-approval for house that we are putting an offer in

    Word of the day

    insolent – adjective (adj) – showing a rude and arrogant lack of respect.

    My strict teacher does not tolerate insolent behavior in her classroom

    What are you grateful for?

    I’m glad I’m writing down what I’m grateful for because its so easy for me to get trapped inside my own brain, my own world, when work (on call specifically) feels like a tornado.

    • Jess being an amazing mom (so patient, so communicative) and leading parenting my example

    Feelings

    • I had lamented and avoided writing a paper at work and realized, after setting ink on paper for just a few minutes, I actually enjoyed the task but had to overcome my own fear of “not knowing enough” or “looking stupid” in front of my colleagues
    • As mentioned above in the summary, I feel good when I can help and serve others. Not in a purely intellectual way, but mentally and emotionally. This is important and perhaps will steer me in a new (or same or different) direction with my career
  • Advanced Operating Systems – OS Structure – Exokernel

    Advanced Operating Systems – OS Structure – Exokernel

    Unlike the SPIN OS Structure, exokernel approaches extensibility by decoupling the authorization to hardware from its actual use, by using something called secure bindings. OS libraries will request secure bindings, the Exokernel returning an encrypted key that grants that particular service to specific resources.

    One other way that the exokernel manages these underlying OS services is via CPU scheduling. The exokernel maintains a linear vector “time slots”, allotting each OS service a certain amount of time to run on the CPU.

    Exokernel also introduces the concept of a STLB (software translation lookaside buffer), the STLB improving performance since, during each process context switch, the exokernel will copy the hardware TLB to a software TLB structure and when the process runs again, the exokernel will copy the software TLB back into the hardware, eliminating the need for a TLB flush.

    Exokernel Approach to Extensibility

    Summary

    Library OS requests access to a specific hardware resource. If access granted, exokernel returns an encrypted key which will be used for future keys

    Examples of Candidate Resources

    Summary

    An OS library will perform some action, sending the Exokernel its encrypted key, and once Exokernel accepts that request, the data processing is cheap. Like packet processing: OS library requests to install predicates for packet filtering; once predicates installed, exokernel will invoke those rules on behalf.

    Implementing Secure Bindings

    Summary

    There are three mechanisms for implementing secure bindings. First is hardware mechanisms, like fetching the TLB. Second would be software, like caching the hardware TLB inside of the OS, avoiding the cost of flushing the TLB during a context switch. And third would be downloading code into the kernel, a feature analogous to SPIN’s approach with logical domains. I don’t really understand these concepts quite yet: it’s all a bit abstract, for now.

    Default Core Services in Exokernel

    Default Core Services in Exokernel

    Summary

    For memory management, the library OS will handle the page fault (uncalled through the register) by presenting a mapping (with the secure binding) to the Exokernel, the Exokernel installing the mapping in the TLB (hardware), this step requiring privileged access.

    Secure Binding

    Summary

    As the instructor put it, this is all a bit dicey. Both SPIN and Exokernel allow library OS privilege for pure performance but how do we guarantee that the insertion of code into the kernel is done … securely? Safely? Will find out soon, probably over the next couple videos

    Memory Management using Software TLB

    Software TLB

    Summary

    In exokernel, during a context switch, the hardware TLB for a process is copied into a STLB (software TLB) and during a context switch, new process’s software TLB will be preloaded by the Exokernel. Of course, if there’s a TLB cache miss, then the standard page fault workflow happens

    Default Core Services in Exokernel (continued)

    Default Core Services continued
    Default Core Services continued

    Summary

    Three is a linear vector of “time slots”, each OS service reserving a slot for its service. If the OS service runs longer than the allotted quantum, the exokernel will punish it, reducing its time slot in the next run. Time is bounded to perform the saving of the context.

    Revocation of Resources

    Summary

    Exokernel can revoke or reclaim permission from a Library OS, the OS sending an up call (and passing a repossession vector) to the Library OS, informing the library OS that it may need to stash away its resources (on disk).

    Putting it all together

    Summary

    While a library OS’s thread is running, an external event will be kicked up to the running process to determine what to do next

    Exokernel Data Structures

    Exokernel Data Structures

    Summary

    Exokernel maintains a data structure for each of the running library OS, maintaining a software TLB that gets loaded during a context switch

    Performance Results of Spin and Exokernel

    Summary

    Absolute numbers are meaningless: it’s the trends that count.