Matt Chung's Site

Author: mattchung

  • Daily Review – Day ending in 2020/09/08

    Daily Review – Day ending in 2020/09/08

    The most interesting side effect of writing these daily reviews is that they stir up meaningful conversations with my wife. Apparently, much of what words I set on paper never make their way out of my mouth (despite thinking that I did verbally share them). Surely this situation of my brain tricking me into thinking that I already communicated something is not unique to me, right?

    This disconnect between what I thought I said and what I actually said makes me wonder how often I fail to articulate the thoughts running through my mind.

    Yesterday

    Writing

    Shared quotes that I captured while watching an inspirational video created by Veritasium, the YouTube video titled “My Life Story”

    Published my notes on “Introduction to Virtualization”

    Best part of my day

    “I have not just a house, but a home, a home filled with love”

    Blinds up means Elliott is awake!

     

    The best part of my day is looking through the living room window and seeing Elliott (bouncing up and down) and Jess cradling her, the two of them standing on the couch and waving at me as I pulled up in the drive way around 7:00 am, around the time when I return from walking the dogs in the morning. Normally, our living room blinds sit shut and we only pull them up during the day when Elliott is awake because our miniature Australian Shepherd tends to bark at every person and crow walking up and down our (busy) hill. So seeing the blinds open as I park the car reminds me that I have not just a house, but a home filled, a home filled with love.

    Graduate School

    Started watching the lecture series on “Shared Memory Machines”. Much of the material is review (from operating systems and compilers and high performance architecture) but I like the feeling of drilling the same material multiple times because every time I revisit these topics, I understand them just a little more, gaining a tad more intuition of computer science.

    Organization

    Dynamo LetraTag – cheap $25.00 label maker that I tossed into the bin

     

    I ended up throwing away the $25.00 label maker that I purchased from Target a couple days ago. The device landed in the bin because the paper cutter kept jamming after every other printed label. Not wanting to waste money, I spent about 10 minutes disassembling the device, unscrewing the mini Phillip screw drivers screwed in on the back, but trying to fix the label maker was a lost cause so I ended up purchasing a higher quality Brother Label Maker that runs about $100.00 (a heavy duty case included).  Sure, it is a bit expensive for a label maker but I believe in paying for quality, paying for items that should last a life time (if they are maintained properly), instead of buying lots of cheap items that get tossed out.

    Work

    Mostly took care of administrative items, like scheduling meetings that I’ve been dragging my feet on. I feel like the first day of the week, I’m normally the most unproductive and have to ease my way into the work week. No exception to yesterday.

    In addition to tackling those low hanging fruit, I met with other teams within AWS to help root cause some packet loss within the EC2 network.

    Today

    Writing

    Publish notes taken from watching yesterday’s lectures (or lectures from a few days ago)

    Organization

    Review OmniFocus’s forecast tab to get a sense of what meetings I have this week and any items that are due soon

    Graduate School

    Wrap up the lectures on “Shared memory” (should take another 30 minutes)

    Start writing some prototype code for collecting virtual memory statistics as part of memory coordinator project (hopefully can squeeze in about an hour of concentrated focus while Jess continues with the night time routine, which starts with Elliott and I bathing in the tub.

    Mental and Physical health

    Attend weekly psychotherapy session. Outside of seeing my therapist (Roy), I realize I rarely talk to anyone (apart from my wife) for an extended period of time. I’m wondering how I will continue these sessions when I move to Renton … we’ll find out in a couple weeks.

    Work

    Operational ticket bash. Our ticket queue is trending towards triple digits so our team will be dedicating the day to closing as many tickets as possible. This is the first ticket bash I don’t expect these dedicated days to become a habit.

    Family

    Tell Jess I love her. Same with Elliott. And the puppies So easy to go throughout the day forgetting how blessed I am. Should do the same thing with the rest of my family members and friends too.

     

  • A snapshot of my understanding before tackling the memory coordinator

    A snapshot of my understanding before tackling the memory coordinator

    Now that I finished writing the vCPU scheduler for project 1, I’m moving on to the second part of the project called the “memory coordinator” and here I’m posting a similar blog post to a snapshot of my understanding of project 1 , the motivation being that I take for granted what I learned throughout graduate school and rarely celebrate these little victories.

    This post will focus on my unknowns and my knowledge gaps as it relates to the memory coordinator that we have to write in C using libvrt. According to project requirements, the memory coordinator should:

    dynamically change the memory of each virtual machine, which will indirectly trigger balloon driver.

    Questions I have

    As I chip away at the project, more questions will inevitably pop up and when they do, I’ll capture them (but in a separate blog post). So here’s my baseline understanding of what a memory coordinator:

    • What are the relevant memory statistics that should be collected?
    • Will the resident set size (RSS) be relevant to the decision making algorithm? Or it is  irrelevant?
    • What is the upper and lower bounds of memory consumption that triggers the ballooning driver to page memory out or page memory in?
    • Will the balloon driver only trigger for virtual machines that are memory constrained?
    • Does the hypervisor’s memory footprint matter (I’m guessing yes, but to what extent)?

     

  • Quotes from “My life story” by Veritasium

    Quotes from “My life story” by Veritasium

    During some down time this evening, I watched the below YouTube video clip filmed and produced by Veritasium and I absolutely loved hearing about his journey, especially about how becoming a father has fundamentally changed the way he views his time. Because I’m in a similar boat: My life looks nothing like it did a little less than a year ago, when my daughter (Elliott) was born. She’s a handful but worth all the sleep the deprivation.  I’m so thankful to have such a wonderful wife who thrives as a mother (seriously: mother of the year award) and has more patience than a meditating monk.

    Anyways, the video “My Life Story” spoke to me and here are some quotes that really resonated:

     I was looking for that well defined path toward a creative career

    I’ve searched for that linear path towards a rewarding and meaningful career and I think to some degree, I’m still searching. Ideally, I could blend my work as a software developer (my dream job just a few years ago) with creative writing.

    “I reached a breaking point .. I was 28 years old and I had spent my whole life up until then building up back up plans, and doing the things that were most likely to succeed …”

    I love learning and love challenging myself and love learning more about the craft of computer science: these are all things that motivate me to pursue my master’s in computer science. But on some level, I know deep down that I want to write and read and teach and mentor: the things that spark joy in my life.

    “Sometimes I think its a blessing not to know how bad you are. If I had known I would have quit. But I didn’t. So I kept working at it

    Working at Amazon for the last 4.5 years, I now realize what it feels like to work side by side with top talent. And had I been fully aware of the gaping holes in my skills and knowledge, I don’t think I would’ve applied to work where I do now.

    I’m all too aware of the survivor bias, that is if you look at the subset of people who are successful at a particular thing, well your kinda ignoring all the experience of the many more people who did not manage to succeed

    Yes yes yes. We often look at all the people who “succeed” and listen to their sage advice of “follow your passion”, overlooking the fact that many many people “follow their passion” and fail to see their dreams manifest.

    Having kids has also made me reevaluate the types of videos I want to be making .. and what I want to be doing with my time

    Although I don’t make videos but as a father, I’m constantly evaluating how I spend my time because I know, deep down, what’s going to matter 10 years or 20 years or 30 years is this: my friends and family. Not fortune or fame (although those things are nice as well).

     

     

  • Introduction to virtualization (notes)

    Introduction to virtualization (notes)

    As system designers, our goal is to design a “black box” system that create an illusion that our users have full and independent access to the underlying hardware of the system. This is merely an abstraction since we are building multi-tenant systems with many applications and many virtual guest machines running on a single piece of hardware, all at the same time.

    To this end, we build what is called a “hypervisor” (code that runs directly on the physical hardware), the software supporting multiple guest machines that run on of it. The guest operating system either be virtualized guest operating system (that has no clue it’s a virtual guest and so that underlying OS binary is the same as it is if you were to install it on a physical server) or be para virtualized operating system (that is aware of the fact that it is virtualized, similar to how the “hosts” in Westworld gain awareness that they are in fact robots)

    Quiz: Virtualization

    Summary

    the concept of virtualization is prolific. We see it in the 60s and 70s when IBM invented the VM/370. We also see it in cloud computing and modern data centers.

    Platform virtualization

    Summary

    As aspiring operating system designers, we want to be able to build the “black box” in which the applications ride on top of, the black box being an illusion of an entire independent hardware system, when really it is not.

    Utility Company

    Summary

    End users resource usage is bursty and we want to amortize the cost of the shared hardware resources. End users have access to large available resources

    Hypervisors

    Hypervisors

    Summary

    Inside the black box, there are two types: native and hosted. Native is bare metal, the hypervisors running directly on the hardware. Hosted, on the other hand, runs as a user application on the Host operating system

    Connecting the dots

    Summary

    Concepts of virtualization date back as far as the 70s, when IBM first invented it with IBM VM 370. Fast following was microkernels, extensibility of OS and SIMOS (late (0s, and then most recently, Xen + VMware (in the 2000s). Now we are looking at virtualizing the data center.

    Full Virtualization

    Full virtualization

    Summary

    With full virtualization, the underlying OS binaries are untouched, no changes required of the OS code itself. To make this work, hypervisor needs to employ some strategies for some system calls that silently fail

    Para Virtualization

    Summary

    Paravirtualization can directly address some of the issues (like silently failing calls) that happens with full virtualization. But OS needs to be modified ; at the same time, can take advantage of optimizations like page coloring.

    Quiz: What percentage of guest OS code may need modification

    Quiz – Guest OS changes < 2%

    Summary

    Less than 2% of code needs modification to support paravirtualization, very minuscule (proof of construction, using Xen)

    Para Virtualization (continued)

    Summary

    With paravirtualization, not many code changes are needed and almost sounds like a no brainer (to me)

  • Daily Review – Day ending in 2020/09/07

    Daily Review – Day ending in 2020/09/07

    Yesterday

    Writing

    Emotions

    • I disassociated and my mind wandered into its “own world” after my wife Jess asked me to watch Elliott later that afternoon.  I think part of the reason why I got so worked up had less to do with her request (cause I actually had a blast with Elliott for that 1 hour) and had to do more with an uneasy feeling caused by my being “stuck” on the school project, which translated to “I have a lot to do with no little time”

    What am I grateful for?

    • The fact that Jess and I can work through conflict and continue as a couple to develop ways to flesh out issues. Our system is not perfect. I like to disconnect and gather my thoughts. She likes to talk issues through and resolve them then and there. We’re meeting somewhere in the middle.

    Physical and Mental Health

    Elliott and daddy and boeing creek park
    • Hiked with the dogs and Elliott (who sat on top of my shoulders most of the walk) and Jess at Boeing creek park, a hidden gem. That park and its thousands of 100 feet ever green trees makes you feel as if you teleported to another world, a world that reminds you how beautiful and how strong nature is. We had witnessed the powerful wind knock down one of the trees and reminded how, as humans, little we are and how we should respect nature

    Music

    • Came up with another beautiful sounding harmony (basically a I-IV-I-V-III-IV progression) that I should capture in my voice notes (will do that shortly)

    Graduate School

    • Finished up the majority of part 1 of my project, writing a scheduler in C that balances virtual CPUs on physical CPUs running on my Xen server. Only piece remaining is adding a “stabilization” feature that does not shuffle CPUs around. This should take maybe another hour of focused attention
    • Read the requirements for part 2 of the project, which sounds like I’ll be emulating a “balloon” driver, a driver that expands and contracts depending on whether the system as a whole needs more or less memory, respectively
    • Finished watching the lengthy video series on “CPU & Device Virtualization”
    • Began watching an even lengthier video series on “Shared Memory Machines”

    Organization

    • Culled all the lingering, unread (and read) e-mails sleeping in my inbox. Amazing how many e-mails that I do not read so I unsubscribed from quite a few newsletters and disabled e-mail forwarding on my undergraduate academic address since I receive nothing but warning notifications around tornado weather

    Work

    • Nothing. At all. Thanks to the three day weekend (appreciate it labor day)

    Friends and Family

    • Finally remembered (thanks to revising my to do list for the day) to video call my Grandma, who hasn’t seen Elliott for a couple months now.
    • Watched Elliott for an hour while Jess ran off to her appointment. Most memorable moment as a father was seeing Elliott stand up (she’s getting so good at holding this position) and with a little assistance (her gently resting her palm in mine) took one of her first steps. Who would’ve known something so little sparks so much joy
    • Packed and taped up our first moving box, the box filled with all sorts of textbooks that were previously sitting on our bookshelf. Oh yea, also off the kitchen table and carried it outside, placing the table in the front yard so that anybody can pick it up (for free). These little moving victories helps me feel as if we are gaining traction and momentum with this major house move

    Miscellaneous

    • Talked on the phone for 45 minutes with our Loan Broker, who did an excellent job describing the different roles and parties involved in a home purchase transaction. So many terms I was unfamiliar with including: the broker, the lender, the underwriter, the title company, the escrow company. More on this on a separate blog post

    Word of the day

    reprisal actions on the part of his troops were brought to his notice in reports made to him

    Definition

    Reprisal – noun – an act of retaliation.

    Today

    Writing

    • Publish post on notes from graduate school and/or a post on how my C skills have sharpened thanks to self studying, graduate school, switching teams at Amazon (2 years ago)

    Organization

    • Review OmniFocus’s forecast tab to get a sense of what meetings I have this week and any items that are due soon

    Graduate School

    • Continue watching lecture on “Shared memory”

    Mental and Physical health

    • Run for 5-10 minutes (not much but these little minutes add up) since I’ve been feeling sluggish lately and tired of seeing my belly begin to sag over my waist (physical activities are just unnecessarily harder when you pack on a few pounds)

    Work

    • Send e-mail to three senior engineers who will be participating in my panel
    • Send meeting invite with Rahul so that they can share their presentation with me, the one that will take place in the first 10 minutes of my panel discussion

    Family

    • Check in with Jess since she’s right: I review my days and plan out the next ones so meticulously but fail to include her (and others) as part of my daily equation.  My argument was that she’s so important and implicitly included as part of my day. But is that true? Not sure if I buy my own excuse so I’ll sit and muse on the idea.
    • Review my list of “Contacts” since I haven’t touched base with many people — friends or family — since COVID-19 hit. Actually, COVID-19 has nothing to do with the fact that I rarely reach out to people. I’m so insular and in my “own world” and it’s easy to forget to reach out to loved ones

     

  • A naive round robin CPU scheduler

    A naive round robin CPU scheduler

    A couple days ago, I spent maybe an hour whipping together a vary naive CPU scheduler for project 1 in advanced operating systems. This naive scheduler pins each of the virtual CPUs in a round robin fashion, not taking utilization (or any other factor) into consideration. For example, say we have four virtual CPUs and two physical CPUs; the scheduler will assign virtual CPU #0 to physical CPU #0, virtual CPU #1 to physical CPU #1, virtual CPU#3 to physical CPU #0 and virtual CPU#0 to physical CPU#1.

    This naive schedule is far from fancy — really the code just performs a mod operation to wrap around the length of the physical CPUs and avoid an index error and carries out a left bit shift operation to populate the bit map — but performs surprisingly well based off monitoring results (below) that measure the utilization of each physical CPU.

    Of course, my final scheduler will pin virtual CPUs to physical CPUS more intelligently,  taking the actual workload (i.e. time in nanoseconds) of the virtual CPUs into consideration.  But as always, I wanted to avoid pre-optimization and jump to some fancy algorithm published in some research paper and I’m glad I started with a primitive scheduler that, for the most part, evenly distributes the work apart from the fifth test (which generates uneven workloads), the only test in which the naive scheduler creates a more uneven workload.

    With this basic prototype in place, I should be able to come up with a more sophisticated algorithm that takes the virtual CPU utilization into consideration.

    Test Case 1

    In this test case, you will run 8 virtual machines that all start pinned to pCPU0. The vCPU of each VM will process the same workload.

    Expected Outcome

    Each pCPU will exhibit an equal balance of vCPUs given the assigned workloads (e.g., if there are 4 pCPUs and 8 vCPUs, then there would be 2 vCPUs per pCPU).

    --------------------------------------------------
0 - usage: 103.0 | mapping ['aos_vm1', 'aos_vm8', 'aos_vm4', 'aos_vm6', 'aos_vm5', 'aos_vm2', 'aos_vm3', 'aos_vm7']
1 - usage: 0.0 | mapping []
2 - usage: 0.0 | mapping []
3 - usage: 0.0 | mapping []
--------------------------------------------------
0 - usage: 99.0 | mapping ['aos_vm1', 'aos_vm8', 'aos_vm4', 'aos_vm6', 'aos_vm5', 'aos_vm2', 'aos_vm3', 'aos_vm7']
1 - usage: 0.0 | mapping []
2 - usage: 0.0 | mapping []
3 - usage: 0.0 | mapping []
--------------------------------------------------
0 - usage: 49.0 | mapping ['aos_vm1', 'aos_vm5']
1 - usage: 47.0 | mapping ['aos_vm8', 'aos_vm2']
2 - usage: 50.0 | mapping ['aos_vm4', 'aos_vm3']
3 - usage: 49.0 | mapping ['aos_vm6', 'aos_vm7']
--------------------------------------------------
0 - usage: 60.0 | mapping ['aos_vm1', 'aos_vm5']
1 - usage: 65.0 | mapping ['aos_vm8', 'aos_vm2']
2 - usage: 61.0 | mapping ['aos_vm4', 'aos_vm3']
3 - usage: 61.0 | mapping ['aos_vm6', 'aos_vm7']
--------------------------------------------------

    Test 2

    In this test case, you will run 8 virtual machines that start with 4 vCPUs pinned to pCPU0 and the other 4 vCPUs pinned to pCPU3. The vCPU of each VM will process the same workload.

    Expected Outcome

    Each pCPU will exhibit an equal balance of vCPUs given the assigned workloads.

    --------------------------------------------------
0 - usage: 102.0 | mapping ['aos_vm1', 'aos_vm4', 'aos_vm5', 'aos_vm3']
1 - usage: 0.0 | mapping []
2 - usage: 0.0 | mapping []
3 - usage: 101.0 | mapping ['aos_vm8', 'aos_vm6', 'aos_vm2', 'aos_vm7']
--------------------------------------------------
0 - usage: 50.0 | mapping ['aos_vm1', 'aos_vm5']
1 - usage: 53.0 | mapping ['aos_vm8', 'aos_vm2']
2 - usage: 51.0 | mapping ['aos_vm4', 'aos_vm3']
3 - usage: 53.0 | mapping ['aos_vm6', 'aos_vm7']
--------------------------------------------------
0 - usage: 102.0 | mapping ['aos_vm1', 'aos_vm5']
1 - usage: 100.0 | mapping ['aos_vm8', 'aos_vm2']
2 - usage: 95.0 | mapping ['aos_vm4', 'aos_vm3']
3 - usage: 99.0 | mapping ['aos_vm6', 'aos_vm7']

    Test Case 3

    In this test case, you will run 8 virtual machines that start with an already balanced mapping of vCPU->pCPU. The vCPU of each VM will process the same workload.

    Expected Outcome

    No vCPU->pCPU mapping changes should occur since a balance state has already been achieved.

    --------------------------------------------------
0 - usage: 63.0 | mapping ['aos_vm1', 'aos_vm5']
1 - usage: 60.0 | mapping ['aos_vm8', 'aos_vm2']
2 - usage: 59.0 | mapping ['aos_vm4', 'aos_vm3']
3 - usage: 58.0 | mapping ['aos_vm6', 'aos_vm7']
--------------------------------------------------
0 - usage: 57.0 | mapping ['aos_vm1', 'aos_vm5']
1 - usage: 60.0 | mapping ['aos_vm8', 'aos_vm2']
2 - usage: 60.0 | mapping ['aos_vm4', 'aos_vm3']
3 - usage: 61.0 | mapping ['aos_vm6', 'aos_vm7']
--------------------------------------------------
0 - usage: 57.0 | mapping ['aos_vm1', 'aos_vm5']
1 - usage: 59.0 | mapping ['aos_vm8', 'aos_vm2']
2 - usage: 59.0 | mapping ['aos_vm4', 'aos_vm3']
3 - usage: 60.0 | mapping ['aos_vm6', 'aos_vm7']

    Test Case 4

    In this test case, you will run 8 virtual machines that start with an equal affinity to each pCPU (i.e., the vCPU of each VM is equally like to run on any pCPU of the host). The vCPU of each VM will process the same workload.

    Expected Outcome

    Each pCPU will exhibit an equal balance of vCPUs given the assigned workloads.

    3 - usage: 60.0 | mapping ['aos_vm3', 'aos_vm7']
--------------------------------------------------
0 - usage: 57.0 | mapping ['aos_vm1', 'aos_vm5']
1 - usage: 61.0 | mapping ['aos_vm8', 'aos_vm2']
2 - usage: 58.0 | mapping ['aos_vm4', 'aos_vm3']
3 - usage: 59.0 | mapping ['aos_vm6', 'aos_vm7']
--------------------------------------------------
0 - usage: 59.0 | mapping ['aos_vm1', 'aos_vm5']
1 - usage: 60.0 | mapping ['aos_vm8', 'aos_vm2']
2 - usage: 60.0 | mapping ['aos_vm4', 'aos_vm3']
3 - usage: 61.0 | mapping ['aos_vm6', 'aos_vm7']
--------------------------------------------------

    Test Case 5

    In this test case, you will run 8 virtual machines that start with an equal affinity to each pCPU (i.e., the vCPU of each VM is equally like to run on any pCPU of the host). Four of these vCPUs will run a heavy workload and the other four vCPUs will run a light workload.

    Expected Outcome

    Each pCPU will exhibit an equal balance of vCPUs given the assigned workloads.

    --------------------------------------------------
0 - usage: 50.0 | mapping ['aos_vm3']
1 - usage: 70.0 | mapping ['aos_vm2', 'aos_vm7']
2 - usage: 142.0 | mapping ['aos_vm1', 'aos_vm4', 'aos_vm6']
3 - usage: 85.0 | mapping ['aos_vm8', 'aos_vm5']
--------------------------------------------------
0 - usage: 88.0 | mapping ['aos_vm1', 'aos_vm7']
1 - usage: 87.0 | mapping ['aos_vm8', 'aos_vm4']
2 - usage: 53.0 | mapping ['aos_vm5']
3 - usage: 119.0 | mapping ['aos_vm6', 'aos_vm2', 'aos_vm3']
--------------------------------------------------
0 - usage: 182.0 | mapping ['aos_vm1', 'aos_vm5', 'aos_vm3', 'aos_vm7']
1 - usage: 36.0 | mapping ['aos_vm8']
2 - usage: 54.0 | mapping ['aos_vm4']
3 - usage: 70.0 | mapping ['aos_vm6', 'aos_vm2']
--------------------------------------------------
0 - usage: 100.0 | mapping ['aos_vm1', 'aos_vm5']
1 - usage: 73.0 | mapping ['aos_vm8', 'aos_vm2']
2 - usage: 99.0 | mapping ['aos_vm4', 'aos_vm3']
3 - usage: 74.0 | mapping ['aos_vm6', 'aos_vm7']
--------------------------------------------------
  • Distancing oneself from a difficult problem

    Distancing oneself from a difficult problem

    I fixed a silly bug just now and wrote working code that can pin multiple virtual CPUs to the physical CPUs.

    Identifying the bug in my code was another classic example of how I needed to distance myself from the problem. Instead of staying up late into the night (late is now 09:30 pm on a weekday) and fixating on debugging the code, I slammed my laptop lid shut and whispered under my breath that I would revisit the problem in the early morning, reassuring myself that stepping away from the problem (in this case, sleeping on it) is often the best — and sometimes only — way to solve a difficult problem.

    There have been countless number of times where I would fixate on a particular piece of code for hours, reading and rereading every line in code, stepping through each instruction with the debugger, only to grow more frustrated that my perseverance failed to pay off. Only when I let the problem simmer in the back of my brain and return to the screen with a fresh new pair of eyes am I able to fix the problem.

    How about you? Do you find distancing yourself from a hard or difficult problem is sometimes the only way to solve it?

    Pnning virtual CPU to physical CPU
  • Weekly Review – Week ending in 2020/09/06

    Weekly Review – Week ending in 2020/09/06

    Writing

    I’m getting much more comfortable with publishing blog posts that are not completely polished. The fear of letting the world see less my less the perfect propose is utter non-sense. In fact, writing and publishing frequently offers two benefits. The first is that the sheer act of writing and setting words on (digital) paper improve my craft. The second benefit of producing words on a regular cadence allows me to track my writing progression, allowing me to critique my writing over time.

    Music

    The trio hanging out at magnuson park

     

    Although I didn’t put in much deliberate practice for neither singing or guitar (apart from practicing a singing the minor scale), I did sing for Elliott almost every night while bathing her, singing “What a wonderful world”. That’s the good stuff, the whole point of developing music skills, right ?

    Graduate School

    Graduate school eats up a good majority of my free time. My studies take place before work, early in the morning, around 04:30 to 05:00 AM, and after work (around 06:00 pm). In total, I get about 2 hours a day, sometimes 3 if I am lucky.

    And when it comes to the advanced operating systems course I am taking right now, there’s never a moment of rest, almost some task to make forward progress on: from reading papers (e.g. “OS Structure – SPIN”), to watching lectures (e.g. “memory virtualization”) to writing code (i.e. a scheduler and memory coordinator).

    But I’m doing my best with the limited amount of time I have and even tracking my progress by publishing my notes from lectures and publishing questions I face and publishing technical problems that I am facing while writing code my project. All of these posts, I hope, will allow me to look back at the end of the semester (about three and a half months away) and feel proud of work I put in and the knowledge I gained.

    Organization

    Current system for organization and time management broke down. The fact that a few items slipped reinforces the fact that the tools (e.g. excel, OmniFocus) do not guarantee organization: it’s the habits and processes. The tools are only a piece of the puzzle, not the solution.

    I had missed a couple appointments and missed a couple important tasks — I hate that feeling. One reason for forgetting about these events is because I failed to book the appointment in my calendar. Another reason is that although some of the events were in my calendar, I didn’t review my calendar and didn’t receive notifications of the event.  Looking back, I can think of a couple ways to fix this. The first is to make sure that for any time sensitive tasks (or tasks with due dates), I need to plug that into my calendar right away. That’s step one. The second step is to enable notifications by configuring the event to notify my phone in advance: 1 week in advanced, 1 day in advanced and then 1 hour in advanced.

    House Organization

    Even though we’re moving into a larger home with more space, I fear that our abysmal home cleaning and organizational skills will follow us (which they will). I’m doing my best to view the dirty home as an opportunity but it’s hard not to feel like my life is spinning out of control when our kitchen looks like this:

    Messy Kitchen (2020-09-05)

    Granted, our lives changed dramatically when our daughter was born … but that was almost a year ago. So that grace period, I think, has passed. The sad reality is that we’re … extremely messy and disorganized. What is it going to take to keep the house in order?

    And as I mentioned in “I’m a messy person: it’s time for a change” post, I’m sick and tired of not my items buried underneath one or another and just overall mountains of junk piling up everywhere throughout the house; this is not the environment in which I want my children to grow up in.

    Physical and Mental Health

    Physical health has taken a back seat and I definitely want to carve out time (even if it is 5 minutes a day) to get my heart pumping. Working from home in the midst of COVID-19 has definitely contributed to lack of exercise for me — some folks have gotten into tip top shape during the lockdowns.

    But I did attend my weekly therapy session and I hope that I can somehow continue seeing my therapist (same person I’ve seen for over four years on a weekly basis) even though I’m moving to Renton, making the commute to his office unsustainable. Let’s see how this all plays out in the about 3 weeks, once my wife and I move.

  • Advanced Operating Systems (Project 1) – monitoring CPU affinity before launching my own scheduler

    Advanced Operating Systems (Project 1) – monitoring CPU affinity before launching my own scheduler

    Project 1 requires that we write a CPU scheduler and memory coordinator. Right now, I’m focusing my attention on the former and the objective for this part of the project is write some C code that pins virtual CPUs to physical CPUs based off of the utilization statistics gathered with the libvrt library (I was able to clear up some of my own confusion by doodling the bitmap data structure passed in as a pointer). We then launch our executable binary and its job is to maximize the utilization across all the physical cores.

    But before launching the scheduler, I want to see what the current scheduler (or lack thereof) is doing in terms of spreading load across the physical CPUs. At a glance, looks like a very naive scheduler (or no scheduler) runs, given that all the virtual guest operating systems are pinned to a single physical CPU:

    0 - usage: 102.0 | mapping ['aos_vm1', 'aos_vm8', 'aos_vm4', 'aos_vm6', 'aos_vm5', 'aos_vm2', 'aos_vm3', 'aos_vm7']
1 - usage: 0.0 | mapping []
2 - usage: 0.0 | mapping []
3 - usage: 0.0 | mapping []
--------------------------------------------------
0 - usage: 100.0 | mapping ['aos_vm1', 'aos_vm8', 'aos_vm4', 'aos_vm6', 'aos_vm5', 'aos_vm2', 'aos_vm3', 'aos_vm7']
1 - usage: 0.0 | mapping []
2 - usage: 0.0 | mapping []
3 - usage: 0.0 | mapping []
--------------------------------------------------
0 - usage: 101.0 | mapping ['aos_vm1', 'aos_vm8', 'aos_vm4', 'aos_vm6', 'aos_vm5', 'aos_vm2', 'aos_vm3', 'aos_vm7']
1 - usage: 0.0 | mapping []
2 - usage: 0.0 | mapping []
3 - usage: 0.0 | mapping []
--------------------------------------------------
0 - usage: 101.0 | mapping ['aos_vm1', 'aos_vm8', 'aos_vm4', 'aos_vm6', 'aos_vm5', 'aos_vm2', 'aos_vm3', 'aos_vm7']
1 - usage: 0.0 | mapping []
2 - usage: 0.0 | mapping []
3 - usage: 0.0 | mapping []
--------------------------------------------------
0 - usage: 102.0 | mapping ['aos_vm1', 'aos_vm8', 'aos_vm4', 'aos_vm6', 'aos_vm5', 'aos_vm2', 'aos_vm3', 'aos_vm7']
1 - usage: 0.0 | mapping []
2 - usage: 0.0 | mapping []
3 - usage: 0.0 | mapping []
--------------------------------------------------
0 - usage: 100.0 | mapping ['aos_vm1', 'aos_vm8', 'aos_vm4', 'aos_vm6', 'aos_vm5', 'aos_vm2', 'aos_vm3', 'aos_vm7']
1 - usage: 0.0 | mapping []
2 - usage: 0.0 | mapping []
3 - usage: 0.0 | mapping [
  • Daily Review – Day ending in 2020/09/04

    Daily Review – Day ending in 2020/09/04

    Yesterday

    Writing

    Music

    • Noodled around on the guitar,  playing chords in a minor key up the neck (need to start structuring and mixing up my practice sessions since they are limited to about 10-15 minutes a day — oh how fatherhood changes how we spend and maximize our alone time)

    Graduate School

    • Hit pause on writing C code and shifted my focus to diagramming the bitmap data structure
    • Finished watching the videos lectures on “memory virtualization” and began watching the next series on CPU & Memory Virtualization
    • Learned the underlying implementation of techniques such as driver ballooning and oblivious page sharing, technologies that I used to apply as a systems administrator — close to 10 years ago — when running a VMWare ESXi cluster. So awesome to be peeling off layers of the onion and gaining a deeper understanding of computer science

    Work

    • Met with the senior network principle engineer to discuss my design document. He agreed that the open questions I had, around failure scenarios, needed to be investigated deeper. All the points he made on data supported approach and scalability and extensibility resonated with me.
    • Ordered a new develop EC2 instance, the new instance costing $110 per month (in comparison to my $25.00 instance). The portal in which I launched the instance warned me that the cost exceeds 75% of most other machines. The way I justify the decision is that I sometimes spend more time in a day watching GCC compile my code, slowing down my development velocity (which costs more than $110.00 per month).

    Friends and Family

    • Miss: Forgot to call my grandma using Facebook Messenger. It’s been several months since she last seen Elliott and she called me the last couple days to see her. Shucks. I even added the item to my Inbox on OmniFocus; I just didn’t reviewed my list. Oh well. Will need to come up some sort of mechanism to periodically review my calendar and my tasks throughout the day.
    • Bathed Elliott last night, despite being completely shattered from waking up at 04:00 (the tiny window of focused work, whenever everyone is sleeping, even the dogs). During our bathes, we started playing this new game: I stretch out my fingers and point at my wedding ring and she tries to slide the ring off and on my finger. This game is an excellent way for her to build her motor skills and she’s improved significantly over the last couple days. It’s really cute to see her eye brows burrow when she struggles with some task and I just always try to encourage her to keep trying (and modify the game slightly to set her up for success).

    Photo of the day

    Mushroom looking damn beautiful

    Today

    As mentioned in my previous posts, I’m moving from North Seattle to Renton (the “burbs”). The entire process moving fast: in less than 3 weeks, we’ll be (fingers crossed) move into our new home that offers more space for Elliott to crawl around. We’re just waiting for the inspection report to land in our inboxes (sometime this morning) so that we can negotiate what repairs (such as replacing the water heater pressure tank) need to happen before we move in.

    Organization

    • Process e-mail inbox down to zero

    Graduate School

    • Wrap up watching lecture on CPU and device scheduling
    • Run the CPU scheduling monitoring code (for Project 1) to get a sense of how the hypervisor’s default scheduler balances work out among the virtual CPUs. I’m guessing that the default behavior of the hypervisor does a decent job of scheduling work so I wonder how much my own scheduler implementation will improve utilization of the underlying physical CPUs

    Mental and Physical health

    • Attend acupuncture session. This will be the first acupuncture session since COVID hit back in February, almost 8 months ago. Our acupuncture closed shop, rightly slow, and only now is starting to open up her clinic.

    Work

    • Revisit the open comments from design review and follow up with AWS Networking teams

    Family

    • Check work calendar and check if I can perform the home inspection at 2:00 PM on Thursday

    What are you grateful for?

    Allergy season ending (no more injecting Flownase every day) and my dogs resting by my feet, providing warm company, as work remotely from home.