Matt Chung's Site

Author: mattchung

  • Quotes from “My life story” by Veritasium

    Quotes from “My life story” by Veritasium

    During some down time this evening, I watched the below YouTube video clip filmed and produced by Veritasium and I absolutely loved hearing about his journey, especially about how becoming a father has fundamentally changed the way he views his time. Because I’m in a similar boat: My life looks nothing like it did a little less than a year ago, when my daughter (Elliott) was born. She’s a handful but worth all the sleep the deprivation.  I’m so thankful to have such a wonderful wife who thrives as a mother (seriously: mother of the year award) and has more patience than a meditating monk.

    Anyways, the video “My Life Story” spoke to me and here are some quotes that really resonated:

     I was looking for that well defined path toward a creative career

    I’ve searched for that linear path towards a rewarding and meaningful career and I think to some degree, I’m still searching. Ideally, I could blend my work as a software developer (my dream job just a few years ago) with creative writing.

    “I reached a breaking point .. I was 28 years old and I had spent my whole life up until then building up back up plans, and doing the things that were most likely to succeed …”

    I love learning and love challenging myself and love learning more about the craft of computer science: these are all things that motivate me to pursue my master’s in computer science. But on some level, I know deep down that I want to write and read and teach and mentor: the things that spark joy in my life.

    “Sometimes I think its a blessing not to know how bad you are. If I had known I would have quit. But I didn’t. So I kept working at it

    Working at Amazon for the last 4.5 years, I now realize what it feels like to work side by side with top talent. And had I been fully aware of the gaping holes in my skills and knowledge, I don’t think I would’ve applied to work where I do now.

    I’m all too aware of the survivor bias, that is if you look at the subset of people who are successful at a particular thing, well your kinda ignoring all the experience of the many more people who did not manage to succeed

    Yes yes yes. We often look at all the people who “succeed” and listen to their sage advice of “follow your passion”, overlooking the fact that many many people “follow their passion” and fail to see their dreams manifest.

    Having kids has also made me reevaluate the types of videos I want to be making .. and what I want to be doing with my time

    Although I don’t make videos but as a father, I’m constantly evaluating how I spend my time because I know, deep down, what’s going to matter 10 years or 20 years or 30 years is this: my friends and family. Not fortune or fame (although those things are nice as well).

     

     

  • Introduction to virtualization (notes)

    Introduction to virtualization (notes)

    As system designers, our goal is to design a “black box” system that create an illusion that our users have full and independent access to the underlying hardware of the system. This is merely an abstraction since we are building multi-tenant systems with many applications and many virtual guest machines running on a single piece of hardware, all at the same time.

    To this end, we build what is called a “hypervisor” (code that runs directly on the physical hardware), the software supporting multiple guest machines that run on of it. The guest operating system either be virtualized guest operating system (that has no clue it’s a virtual guest and so that underlying OS binary is the same as it is if you were to install it on a physical server) or be para virtualized operating system (that is aware of the fact that it is virtualized, similar to how the “hosts” in Westworld gain awareness that they are in fact robots)

    Quiz: Virtualization

    Summary

    the concept of virtualization is prolific. We see it in the 60s and 70s when IBM invented the VM/370. We also see it in cloud computing and modern data centers.

    Platform virtualization

    Summary

    As aspiring operating system designers, we want to be able to build the “black box” in which the applications ride on top of, the black box being an illusion of an entire independent hardware system, when really it is not.

    Utility Company

    Summary

    End users resource usage is bursty and we want to amortize the cost of the shared hardware resources. End users have access to large available resources

    Hypervisors

    Hypervisors

    Summary

    Inside the black box, there are two types: native and hosted. Native is bare metal, the hypervisors running directly on the hardware. Hosted, on the other hand, runs as a user application on the Host operating system

    Connecting the dots

    Summary

    Concepts of virtualization date back as far as the 70s, when IBM first invented it with IBM VM 370. Fast following was microkernels, extensibility of OS and SIMOS (late (0s, and then most recently, Xen + VMware (in the 2000s). Now we are looking at virtualizing the data center.

    Full Virtualization

    Full virtualization

    Summary

    With full virtualization, the underlying OS binaries are untouched, no changes required of the OS code itself. To make this work, hypervisor needs to employ some strategies for some system calls that silently fail

    Para Virtualization

    Summary

    Paravirtualization can directly address some of the issues (like silently failing calls) that happens with full virtualization. But OS needs to be modified ; at the same time, can take advantage of optimizations like page coloring.

    Quiz: What percentage of guest OS code may need modification

    Quiz – Guest OS changes < 2%

    Summary

    Less than 2% of code needs modification to support paravirtualization, very minuscule (proof of construction, using Xen)

    Para Virtualization (continued)

    Summary

    With paravirtualization, not many code changes are needed and almost sounds like a no brainer (to me)

  • Daily Review – Day ending in 2020/09/07

    Daily Review – Day ending in 2020/09/07

    Yesterday

    Writing

    Emotions

    • I disassociated and my mind wandered into its “own world” after my wife Jess asked me to watch Elliott later that afternoon.  I think part of the reason why I got so worked up had less to do with her request (cause I actually had a blast with Elliott for that 1 hour) and had to do more with an uneasy feeling caused by my being “stuck” on the school project, which translated to “I have a lot to do with no little time”

    What am I grateful for?

    • The fact that Jess and I can work through conflict and continue as a couple to develop ways to flesh out issues. Our system is not perfect. I like to disconnect and gather my thoughts. She likes to talk issues through and resolve them then and there. We’re meeting somewhere in the middle.

    Physical and Mental Health

    Elliott and daddy and boeing creek park
    • Hiked with the dogs and Elliott (who sat on top of my shoulders most of the walk) and Jess at Boeing creek park, a hidden gem. That park and its thousands of 100 feet ever green trees makes you feel as if you teleported to another world, a world that reminds you how beautiful and how strong nature is. We had witnessed the powerful wind knock down one of the trees and reminded how, as humans, little we are and how we should respect nature

    Music

    • Came up with another beautiful sounding harmony (basically a I-IV-I-V-III-IV progression) that I should capture in my voice notes (will do that shortly)

    Graduate School

    • Finished up the majority of part 1 of my project, writing a scheduler in C that balances virtual CPUs on physical CPUs running on my Xen server. Only piece remaining is adding a “stabilization” feature that does not shuffle CPUs around. This should take maybe another hour of focused attention
    • Read the requirements for part 2 of the project, which sounds like I’ll be emulating a “balloon” driver, a driver that expands and contracts depending on whether the system as a whole needs more or less memory, respectively
    • Finished watching the lengthy video series on “CPU & Device Virtualization”
    • Began watching an even lengthier video series on “Shared Memory Machines”

    Organization

    • Culled all the lingering, unread (and read) e-mails sleeping in my inbox. Amazing how many e-mails that I do not read so I unsubscribed from quite a few newsletters and disabled e-mail forwarding on my undergraduate academic address since I receive nothing but warning notifications around tornado weather

    Work

    • Nothing. At all. Thanks to the three day weekend (appreciate it labor day)

    Friends and Family

    • Finally remembered (thanks to revising my to do list for the day) to video call my Grandma, who hasn’t seen Elliott for a couple months now.
    • Watched Elliott for an hour while Jess ran off to her appointment. Most memorable moment as a father was seeing Elliott stand up (she’s getting so good at holding this position) and with a little assistance (her gently resting her palm in mine) took one of her first steps. Who would’ve known something so little sparks so much joy
    • Packed and taped up our first moving box, the box filled with all sorts of textbooks that were previously sitting on our bookshelf. Oh yea, also off the kitchen table and carried it outside, placing the table in the front yard so that anybody can pick it up (for free). These little moving victories helps me feel as if we are gaining traction and momentum with this major house move

    Miscellaneous

    • Talked on the phone for 45 minutes with our Loan Broker, who did an excellent job describing the different roles and parties involved in a home purchase transaction. So many terms I was unfamiliar with including: the broker, the lender, the underwriter, the title company, the escrow company. More on this on a separate blog post

    Word of the day

    reprisal actions on the part of his troops were brought to his notice in reports made to him

    Definition

    Reprisal – noun – an act of retaliation.

    Today

    Writing

    • Publish post on notes from graduate school and/or a post on how my C skills have sharpened thanks to self studying, graduate school, switching teams at Amazon (2 years ago)

    Organization

    • Review OmniFocus’s forecast tab to get a sense of what meetings I have this week and any items that are due soon

    Graduate School

    • Continue watching lecture on “Shared memory”

    Mental and Physical health

    • Run for 5-10 minutes (not much but these little minutes add up) since I’ve been feeling sluggish lately and tired of seeing my belly begin to sag over my waist (physical activities are just unnecessarily harder when you pack on a few pounds)

    Work

    • Send e-mail to three senior engineers who will be participating in my panel
    • Send meeting invite with Rahul so that they can share their presentation with me, the one that will take place in the first 10 minutes of my panel discussion

    Family

    • Check in with Jess since she’s right: I review my days and plan out the next ones so meticulously but fail to include her (and others) as part of my daily equation.  My argument was that she’s so important and implicitly included as part of my day. But is that true? Not sure if I buy my own excuse so I’ll sit and muse on the idea.
    • Review my list of “Contacts” since I haven’t touched base with many people — friends or family — since COVID-19 hit. Actually, COVID-19 has nothing to do with the fact that I rarely reach out to people. I’m so insular and in my “own world” and it’s easy to forget to reach out to loved ones

     

  • A naive round robin CPU scheduler

    A naive round robin CPU scheduler

    A couple days ago, I spent maybe an hour whipping together a vary naive CPU scheduler for project 1 in advanced operating systems. This naive scheduler pins each of the virtual CPUs in a round robin fashion, not taking utilization (or any other factor) into consideration. For example, say we have four virtual CPUs and two physical CPUs; the scheduler will assign virtual CPU #0 to physical CPU #0, virtual CPU #1 to physical CPU #1, virtual CPU#3 to physical CPU #0 and virtual CPU#0 to physical CPU#1.

    This naive schedule is far from fancy — really the code just performs a mod operation to wrap around the length of the physical CPUs and avoid an index error and carries out a left bit shift operation to populate the bit map — but performs surprisingly well based off monitoring results (below) that measure the utilization of each physical CPU.

    Of course, my final scheduler will pin virtual CPUs to physical CPUS more intelligently,  taking the actual workload (i.e. time in nanoseconds) of the virtual CPUs into consideration.  But as always, I wanted to avoid pre-optimization and jump to some fancy algorithm published in some research paper and I’m glad I started with a primitive scheduler that, for the most part, evenly distributes the work apart from the fifth test (which generates uneven workloads), the only test in which the naive scheduler creates a more uneven workload.

    With this basic prototype in place, I should be able to come up with a more sophisticated algorithm that takes the virtual CPU utilization into consideration.

    Test Case 1

    In this test case, you will run 8 virtual machines that all start pinned to pCPU0. The vCPU of each VM will process the same workload.

    Expected Outcome

    Each pCPU will exhibit an equal balance of vCPUs given the assigned workloads (e.g., if there are 4 pCPUs and 8 vCPUs, then there would be 2 vCPUs per pCPU).

    --------------------------------------------------
0 - usage: 103.0 | mapping ['aos_vm1', 'aos_vm8', 'aos_vm4', 'aos_vm6', 'aos_vm5', 'aos_vm2', 'aos_vm3', 'aos_vm7']
1 - usage: 0.0 | mapping []
2 - usage: 0.0 | mapping []
3 - usage: 0.0 | mapping []
--------------------------------------------------
0 - usage: 99.0 | mapping ['aos_vm1', 'aos_vm8', 'aos_vm4', 'aos_vm6', 'aos_vm5', 'aos_vm2', 'aos_vm3', 'aos_vm7']
1 - usage: 0.0 | mapping []
2 - usage: 0.0 | mapping []
3 - usage: 0.0 | mapping []
--------------------------------------------------
0 - usage: 49.0 | mapping ['aos_vm1', 'aos_vm5']
1 - usage: 47.0 | mapping ['aos_vm8', 'aos_vm2']
2 - usage: 50.0 | mapping ['aos_vm4', 'aos_vm3']
3 - usage: 49.0 | mapping ['aos_vm6', 'aos_vm7']
--------------------------------------------------
0 - usage: 60.0 | mapping ['aos_vm1', 'aos_vm5']
1 - usage: 65.0 | mapping ['aos_vm8', 'aos_vm2']
2 - usage: 61.0 | mapping ['aos_vm4', 'aos_vm3']
3 - usage: 61.0 | mapping ['aos_vm6', 'aos_vm7']
--------------------------------------------------

    Test 2

    In this test case, you will run 8 virtual machines that start with 4 vCPUs pinned to pCPU0 and the other 4 vCPUs pinned to pCPU3. The vCPU of each VM will process the same workload.

    Expected Outcome

    Each pCPU will exhibit an equal balance of vCPUs given the assigned workloads.

    --------------------------------------------------
0 - usage: 102.0 | mapping ['aos_vm1', 'aos_vm4', 'aos_vm5', 'aos_vm3']
1 - usage: 0.0 | mapping []
2 - usage: 0.0 | mapping []
3 - usage: 101.0 | mapping ['aos_vm8', 'aos_vm6', 'aos_vm2', 'aos_vm7']
--------------------------------------------------
0 - usage: 50.0 | mapping ['aos_vm1', 'aos_vm5']
1 - usage: 53.0 | mapping ['aos_vm8', 'aos_vm2']
2 - usage: 51.0 | mapping ['aos_vm4', 'aos_vm3']
3 - usage: 53.0 | mapping ['aos_vm6', 'aos_vm7']
--------------------------------------------------
0 - usage: 102.0 | mapping ['aos_vm1', 'aos_vm5']
1 - usage: 100.0 | mapping ['aos_vm8', 'aos_vm2']
2 - usage: 95.0 | mapping ['aos_vm4', 'aos_vm3']
3 - usage: 99.0 | mapping ['aos_vm6', 'aos_vm7']

    Test Case 3

    In this test case, you will run 8 virtual machines that start with an already balanced mapping of vCPU->pCPU. The vCPU of each VM will process the same workload.

    Expected Outcome

    No vCPU->pCPU mapping changes should occur since a balance state has already been achieved.

    --------------------------------------------------
0 - usage: 63.0 | mapping ['aos_vm1', 'aos_vm5']
1 - usage: 60.0 | mapping ['aos_vm8', 'aos_vm2']
2 - usage: 59.0 | mapping ['aos_vm4', 'aos_vm3']
3 - usage: 58.0 | mapping ['aos_vm6', 'aos_vm7']
--------------------------------------------------
0 - usage: 57.0 | mapping ['aos_vm1', 'aos_vm5']
1 - usage: 60.0 | mapping ['aos_vm8', 'aos_vm2']
2 - usage: 60.0 | mapping ['aos_vm4', 'aos_vm3']
3 - usage: 61.0 | mapping ['aos_vm6', 'aos_vm7']
--------------------------------------------------
0 - usage: 57.0 | mapping ['aos_vm1', 'aos_vm5']
1 - usage: 59.0 | mapping ['aos_vm8', 'aos_vm2']
2 - usage: 59.0 | mapping ['aos_vm4', 'aos_vm3']
3 - usage: 60.0 | mapping ['aos_vm6', 'aos_vm7']

    Test Case 4

    In this test case, you will run 8 virtual machines that start with an equal affinity to each pCPU (i.e., the vCPU of each VM is equally like to run on any pCPU of the host). The vCPU of each VM will process the same workload.

    Expected Outcome

    Each pCPU will exhibit an equal balance of vCPUs given the assigned workloads.

    3 - usage: 60.0 | mapping ['aos_vm3', 'aos_vm7']
--------------------------------------------------
0 - usage: 57.0 | mapping ['aos_vm1', 'aos_vm5']
1 - usage: 61.0 | mapping ['aos_vm8', 'aos_vm2']
2 - usage: 58.0 | mapping ['aos_vm4', 'aos_vm3']
3 - usage: 59.0 | mapping ['aos_vm6', 'aos_vm7']
--------------------------------------------------
0 - usage: 59.0 | mapping ['aos_vm1', 'aos_vm5']
1 - usage: 60.0 | mapping ['aos_vm8', 'aos_vm2']
2 - usage: 60.0 | mapping ['aos_vm4', 'aos_vm3']
3 - usage: 61.0 | mapping ['aos_vm6', 'aos_vm7']
--------------------------------------------------

    Test Case 5

    In this test case, you will run 8 virtual machines that start with an equal affinity to each pCPU (i.e., the vCPU of each VM is equally like to run on any pCPU of the host). Four of these vCPUs will run a heavy workload and the other four vCPUs will run a light workload.

    Expected Outcome

    Each pCPU will exhibit an equal balance of vCPUs given the assigned workloads.

    --------------------------------------------------
0 - usage: 50.0 | mapping ['aos_vm3']
1 - usage: 70.0 | mapping ['aos_vm2', 'aos_vm7']
2 - usage: 142.0 | mapping ['aos_vm1', 'aos_vm4', 'aos_vm6']
3 - usage: 85.0 | mapping ['aos_vm8', 'aos_vm5']
--------------------------------------------------
0 - usage: 88.0 | mapping ['aos_vm1', 'aos_vm7']
1 - usage: 87.0 | mapping ['aos_vm8', 'aos_vm4']
2 - usage: 53.0 | mapping ['aos_vm5']
3 - usage: 119.0 | mapping ['aos_vm6', 'aos_vm2', 'aos_vm3']
--------------------------------------------------
0 - usage: 182.0 | mapping ['aos_vm1', 'aos_vm5', 'aos_vm3', 'aos_vm7']
1 - usage: 36.0 | mapping ['aos_vm8']
2 - usage: 54.0 | mapping ['aos_vm4']
3 - usage: 70.0 | mapping ['aos_vm6', 'aos_vm2']
--------------------------------------------------
0 - usage: 100.0 | mapping ['aos_vm1', 'aos_vm5']
1 - usage: 73.0 | mapping ['aos_vm8', 'aos_vm2']
2 - usage: 99.0 | mapping ['aos_vm4', 'aos_vm3']
3 - usage: 74.0 | mapping ['aos_vm6', 'aos_vm7']
--------------------------------------------------
  • Distancing oneself from a difficult problem

    Distancing oneself from a difficult problem

    I fixed a silly bug just now and wrote working code that can pin multiple virtual CPUs to the physical CPUs.

    Identifying the bug in my code was another classic example of how I needed to distance myself from the problem. Instead of staying up late into the night (late is now 09:30 pm on a weekday) and fixating on debugging the code, I slammed my laptop lid shut and whispered under my breath that I would revisit the problem in the early morning, reassuring myself that stepping away from the problem (in this case, sleeping on it) is often the best — and sometimes only — way to solve a difficult problem.

    There have been countless number of times where I would fixate on a particular piece of code for hours, reading and rereading every line in code, stepping through each instruction with the debugger, only to grow more frustrated that my perseverance failed to pay off. Only when I let the problem simmer in the back of my brain and return to the screen with a fresh new pair of eyes am I able to fix the problem.

    How about you? Do you find distancing yourself from a hard or difficult problem is sometimes the only way to solve it?

    Pnning virtual CPU to physical CPU
  • Weekly Review – Week ending in 2020/09/06

    Weekly Review – Week ending in 2020/09/06

    Writing

    I’m getting much more comfortable with publishing blog posts that are not completely polished. The fear of letting the world see less my less the perfect propose is utter non-sense. In fact, writing and publishing frequently offers two benefits. The first is that the sheer act of writing and setting words on (digital) paper improve my craft. The second benefit of producing words on a regular cadence allows me to track my writing progression, allowing me to critique my writing over time.

    Music

    The trio hanging out at magnuson park

     

    Although I didn’t put in much deliberate practice for neither singing or guitar (apart from practicing a singing the minor scale), I did sing for Elliott almost every night while bathing her, singing “What a wonderful world”. That’s the good stuff, the whole point of developing music skills, right ?

    Graduate School

    Graduate school eats up a good majority of my free time. My studies take place before work, early in the morning, around 04:30 to 05:00 AM, and after work (around 06:00 pm). In total, I get about 2 hours a day, sometimes 3 if I am lucky.

    And when it comes to the advanced operating systems course I am taking right now, there’s never a moment of rest, almost some task to make forward progress on: from reading papers (e.g. “OS Structure – SPIN”), to watching lectures (e.g. “memory virtualization”) to writing code (i.e. a scheduler and memory coordinator).

    But I’m doing my best with the limited amount of time I have and even tracking my progress by publishing my notes from lectures and publishing questions I face and publishing technical problems that I am facing while writing code my project. All of these posts, I hope, will allow me to look back at the end of the semester (about three and a half months away) and feel proud of work I put in and the knowledge I gained.

    Organization

    Current system for organization and time management broke down. The fact that a few items slipped reinforces the fact that the tools (e.g. excel, OmniFocus) do not guarantee organization: it’s the habits and processes. The tools are only a piece of the puzzle, not the solution.

    I had missed a couple appointments and missed a couple important tasks — I hate that feeling. One reason for forgetting about these events is because I failed to book the appointment in my calendar. Another reason is that although some of the events were in my calendar, I didn’t review my calendar and didn’t receive notifications of the event.  Looking back, I can think of a couple ways to fix this. The first is to make sure that for any time sensitive tasks (or tasks with due dates), I need to plug that into my calendar right away. That’s step one. The second step is to enable notifications by configuring the event to notify my phone in advance: 1 week in advanced, 1 day in advanced and then 1 hour in advanced.

    House Organization

    Even though we’re moving into a larger home with more space, I fear that our abysmal home cleaning and organizational skills will follow us (which they will). I’m doing my best to view the dirty home as an opportunity but it’s hard not to feel like my life is spinning out of control when our kitchen looks like this:

    Messy Kitchen (2020-09-05)

    Granted, our lives changed dramatically when our daughter was born … but that was almost a year ago. So that grace period, I think, has passed. The sad reality is that we’re … extremely messy and disorganized. What is it going to take to keep the house in order?

    And as I mentioned in “I’m a messy person: it’s time for a change” post, I’m sick and tired of not my items buried underneath one or another and just overall mountains of junk piling up everywhere throughout the house; this is not the environment in which I want my children to grow up in.

    Physical and Mental Health

    Physical health has taken a back seat and I definitely want to carve out time (even if it is 5 minutes a day) to get my heart pumping. Working from home in the midst of COVID-19 has definitely contributed to lack of exercise for me — some folks have gotten into tip top shape during the lockdowns.

    But I did attend my weekly therapy session and I hope that I can somehow continue seeing my therapist (same person I’ve seen for over four years on a weekly basis) even though I’m moving to Renton, making the commute to his office unsustainable. Let’s see how this all plays out in the about 3 weeks, once my wife and I move.

  • Advanced Operating Systems (Project 1) – monitoring CPU affinity before launching my own scheduler

    Advanced Operating Systems (Project 1) – monitoring CPU affinity before launching my own scheduler

    Project 1 requires that we write a CPU scheduler and memory coordinator. Right now, I’m focusing my attention on the former and the objective for this part of the project is write some C code that pins virtual CPUs to physical CPUs based off of the utilization statistics gathered with the libvrt library (I was able to clear up some of my own confusion by doodling the bitmap data structure passed in as a pointer). We then launch our executable binary and its job is to maximize the utilization across all the physical cores.

    But before launching the scheduler, I want to see what the current scheduler (or lack thereof) is doing in terms of spreading load across the physical CPUs. At a glance, looks like a very naive scheduler (or no scheduler) runs, given that all the virtual guest operating systems are pinned to a single physical CPU:

    0 - usage: 102.0 | mapping ['aos_vm1', 'aos_vm8', 'aos_vm4', 'aos_vm6', 'aos_vm5', 'aos_vm2', 'aos_vm3', 'aos_vm7']
1 - usage: 0.0 | mapping []
2 - usage: 0.0 | mapping []
3 - usage: 0.0 | mapping []
--------------------------------------------------
0 - usage: 100.0 | mapping ['aos_vm1', 'aos_vm8', 'aos_vm4', 'aos_vm6', 'aos_vm5', 'aos_vm2', 'aos_vm3', 'aos_vm7']
1 - usage: 0.0 | mapping []
2 - usage: 0.0 | mapping []
3 - usage: 0.0 | mapping []
--------------------------------------------------
0 - usage: 101.0 | mapping ['aos_vm1', 'aos_vm8', 'aos_vm4', 'aos_vm6', 'aos_vm5', 'aos_vm2', 'aos_vm3', 'aos_vm7']
1 - usage: 0.0 | mapping []
2 - usage: 0.0 | mapping []
3 - usage: 0.0 | mapping []
--------------------------------------------------
0 - usage: 101.0 | mapping ['aos_vm1', 'aos_vm8', 'aos_vm4', 'aos_vm6', 'aos_vm5', 'aos_vm2', 'aos_vm3', 'aos_vm7']
1 - usage: 0.0 | mapping []
2 - usage: 0.0 | mapping []
3 - usage: 0.0 | mapping []
--------------------------------------------------
0 - usage: 102.0 | mapping ['aos_vm1', 'aos_vm8', 'aos_vm4', 'aos_vm6', 'aos_vm5', 'aos_vm2', 'aos_vm3', 'aos_vm7']
1 - usage: 0.0 | mapping []
2 - usage: 0.0 | mapping []
3 - usage: 0.0 | mapping []
--------------------------------------------------
0 - usage: 100.0 | mapping ['aos_vm1', 'aos_vm8', 'aos_vm4', 'aos_vm6', 'aos_vm5', 'aos_vm2', 'aos_vm3', 'aos_vm7']
1 - usage: 0.0 | mapping []
2 - usage: 0.0 | mapping []
3 - usage: 0.0 | mapping [
  • Daily Review – Day ending in 2020/09/04

    Daily Review – Day ending in 2020/09/04

    Yesterday

    Writing

    Music

    • Noodled around on the guitar,  playing chords in a minor key up the neck (need to start structuring and mixing up my practice sessions since they are limited to about 10-15 minutes a day — oh how fatherhood changes how we spend and maximize our alone time)

    Graduate School

    • Hit pause on writing C code and shifted my focus to diagramming the bitmap data structure
    • Finished watching the videos lectures on “memory virtualization” and began watching the next series on CPU & Memory Virtualization
    • Learned the underlying implementation of techniques such as driver ballooning and oblivious page sharing, technologies that I used to apply as a systems administrator — close to 10 years ago — when running a VMWare ESXi cluster. So awesome to be peeling off layers of the onion and gaining a deeper understanding of computer science

    Work

    • Met with the senior network principle engineer to discuss my design document. He agreed that the open questions I had, around failure scenarios, needed to be investigated deeper. All the points he made on data supported approach and scalability and extensibility resonated with me.
    • Ordered a new develop EC2 instance, the new instance costing $110 per month (in comparison to my $25.00 instance). The portal in which I launched the instance warned me that the cost exceeds 75% of most other machines. The way I justify the decision is that I sometimes spend more time in a day watching GCC compile my code, slowing down my development velocity (which costs more than $110.00 per month).

    Friends and Family

    • Miss: Forgot to call my grandma using Facebook Messenger. It’s been several months since she last seen Elliott and she called me the last couple days to see her. Shucks. I even added the item to my Inbox on OmniFocus; I just didn’t reviewed my list. Oh well. Will need to come up some sort of mechanism to periodically review my calendar and my tasks throughout the day.
    • Bathed Elliott last night, despite being completely shattered from waking up at 04:00 (the tiny window of focused work, whenever everyone is sleeping, even the dogs). During our bathes, we started playing this new game: I stretch out my fingers and point at my wedding ring and she tries to slide the ring off and on my finger. This game is an excellent way for her to build her motor skills and she’s improved significantly over the last couple days. It’s really cute to see her eye brows burrow when she struggles with some task and I just always try to encourage her to keep trying (and modify the game slightly to set her up for success).

    Photo of the day

    Mushroom looking damn beautiful

    Today

    As mentioned in my previous posts, I’m moving from North Seattle to Renton (the “burbs”). The entire process moving fast: in less than 3 weeks, we’ll be (fingers crossed) move into our new home that offers more space for Elliott to crawl around. We’re just waiting for the inspection report to land in our inboxes (sometime this morning) so that we can negotiate what repairs (such as replacing the water heater pressure tank) need to happen before we move in.

    Organization

    • Process e-mail inbox down to zero

    Graduate School

    • Wrap up watching lecture on CPU and device scheduling
    • Run the CPU scheduling monitoring code (for Project 1) to get a sense of how the hypervisor’s default scheduler balances work out among the virtual CPUs. I’m guessing that the default behavior of the hypervisor does a decent job of scheduling work so I wonder how much my own scheduler implementation will improve utilization of the underlying physical CPUs

    Mental and Physical health

    • Attend acupuncture session. This will be the first acupuncture session since COVID hit back in February, almost 8 months ago. Our acupuncture closed shop, rightly slow, and only now is starting to open up her clinic.

    Work

    • Revisit the open comments from design review and follow up with AWS Networking teams

    Family

    • Check work calendar and check if I can perform the home inspection at 2:00 PM on Thursday

    What are you grateful for?

    Allergy season ending (no more injecting Flownase every day) and my dogs resting by my feet, providing warm company, as work remotely from home.

  • Making sense of libvrt bit map when calling virDomainPinVcpu

    Making sense of libvrt bit map when calling virDomainPinVcpu

    On my iPad this morning, I doodled the above figure to help me better understand how I should be calling the function virDomainPinVcpu (as part of project 1 for my advanced operating systems course).  The function requires two parameters which I found a bit confusing: a pointer to the cpu map (i.e. bit map) and length of the map itself.

    In order to generate the CPU map, we need a couple pieces of information: the number of physical CPUs and the number of virtual CPUs (for a guest virtual machine).  This information can be obtained by calling virNodeGetInfo and virDomainGetVcpus, respectively1.

    First, you need to get the number of byes needed to represent the physical CPUs. Let’s say our hypervisor runs 8 physical CPUs. In this case, we’d need a single byte: bit 0 represents CPU#0, bit 1 represents CPU#1 … bit 7 represents CPU#7. If we had 16 physical CPUs, then we’d need 2 bytes. Let’s call this physical_cpu_ptr.

    Then, each virtual CPU will contain a mapping to its own unique physical_cpu_ptr.

    Example

    Below is an output from within GDB. In this example, my physical hypervisor has CPUs and each virtual machine runs 2 virtual CPUs.  Based off of what I mentioned above, that means mean the virtual machine’s bitmap contains 2 bytes. First byte is for virtual CPU#1 and second byte is for virtual CPU#2 — can run on any one of the four physical CPUs. Hence why only the first four bits are set.

    (gdb) x /2b cpu_maps
0x555555772710: 0x0f 0x0f

    References

    1 – https://stackoverflow.com/questions/46254411/what-is-the-argument-cpumaps-and-maplen-in-api-virdomaingetvcpus-of-libvirt

  • Daily Review – Day ending in 2020/09/03

    Daily Review – Day ending in 2020/09/03

    Writing

    Mental and Physical Health

    • At the park, ran around in circles while holding Elliott in my arms … that sort of counts as exercise, right?

    Music

    • Nothing at all

    Graduate School

    • Migrated from my Virtualbox environment to Digital Ocean. My local Ubuntu instance kept crashing, the KVM service unable to start up the guest operating systems. The instances would just hang for no reason and instead of wasting time troubleshooting, I’d rather focus on the code itself
    • Wrestled with libvrt’s documentation and finally was able to collect CPU statistics using the API (see snippet below). The documentation sometimes says you need to pass in a struct when they mean pointer (and vice versa)
      
    cpuTime: 119900000000
root@aos-kvm-ubuntu-18:/tmp# ./vcpu-scheduler 
Num domains: 2
Active Domain IDs:
2
1
State: 1, nrVirtCpu: 2, cpuTime=311380000000
Domain CPU Stats:
cpuTime: 162140000000
cpuTime: 144030000000
State: 1, nrVirtCpu: 2, cpuTime=268990000000
Domain CPU Stats:
cpuTime: 142960000000
cpuTime: 119900000000

    Work

    • Firmed up my design document at work, addressing other engineer’s feedback that was sprinkled in the Quip document
    • E-mailed Asians@ so that they include my upcoming event in the newsletter

    Organization

    • Using my ScanSnap IX1500, I scanned all my loose documents (e.g. tax statements, W-2 copies, etc) and then shredded them, clearing up space on my desk and just overall getting a little more organization

    Friends and Family

    Elliott getting stuck in 3 tier shelf
    • Popped into Wells Fargo Bank to transfer earnest money to the escrow company. I had originally tried calling the bank at around 04:00 AM (when their support center opened) to have both enable wire transfers on my account and bump up the transfer limit but quickly realized that sending tens of thousands of dollars should be probably be done in person at the branch
    • Unexpectedly watched Elliott for the about 3 hours yesterday. Jess was already running behind for the inspection for our new home located in Renton and Elliott was still napping. So instead of waking her up, she left her with me in the middle of the work day and now I truly understand the plight of parents all around the world, parents who need to work from home while taking care of young children: it’s impossible.
    • Scheduled follow up veterinarian appointments for both Metric and Mushroom. Metric’s ear developed a little fissure in her ear, looking as if a tick made a home and then evacuated. And on the back of mushroom’s neck, there’s a maybe 5 or 6 white head looking bumps that are located in the place where I had applied her flea medication a couple weeks ago

    Today

    Organization

    • Call Well’s Fargo (again) and see if they can bump the limit on my wire transfer since I want to avoid going to a Wells Fargo branch due to COVID-19
    • Migrate straggling sticky notes that are sitting on desk and process them into “Writing Ideas” or “Inbox” in Omnifocus

    Graduate School

    • Finish memory virtualization series
    • Write a few more lines of C code to get a better sense of how I’m going to write the scheduler (all a bit fuzzy right now). Not sure how scheduler is going to integrate with KVM and not sure what algorithms I’ll select and implement

    Work

    • Meet with principle network engineer to close out open questions on my design document
    • Attend weekly operations meeting and discuss events that popped up throughout the week

    Family

    • Carry out morning routine: walk with Elliott and Jess and the dogs, blend up a delicious strawberry and blueberry and banana smoothie (thanks to Jess, who picked up the necessary ingredients from Trader Joe’s yesterday), feed the dogs their raw food
    • Reply to loan broker and see if they can lock in the 2.875% rate for our mortgage
    • Hold a discussion with Shuk (our realtor) and finalize what issues that popped up during home inspection need to get fixed before we move forward into the next step

    What are you grateful for?

    Spending 1 on 1 time with little Elliott yesterday (despite needing to watch her last minute during a work day). I cannot really explain it but I think I’m developing some sort of paternal love for my daughter, feelings that I’ve never experienced before, not for any human and not for any of my beloved pets. I cannot really describe the emotion … but she’s able to put a wrinkle in my nose just by sitting there and shooting me a smile.

    Feelings

    After 3 hours of watching Elliott yesterday, I was wiped out, taxed physically and emotionally. That little 11 month year old crawls faster than a rattlesnake and I’m having a hard time keeping up with her. She puts everything in her mouth, from dog hair to shreds of paper (but I would like to call out a victory from yesterday, a proud dad moment, when I was able to free a piece of paper from her mouth before she swallowed it: dad goals).