Matt Chung's Site

Tag: on call operations

Daily Review – Day ending in 2020/09/03
Writing
- Blogged and published an entry for my daily review
- Blogged and published an entry on L3 Microkernel
Mental and Physical Health
- At the park, ran around in circles while holding Elliott in my arms … that sort of counts as exercise, right?
Music
- Nothing at all
Graduate School
- Migrated from my Virtualbox environment to Digital Ocean. My local Ubuntu instance kept crashing, the KVM service unable to start up the guest operating systems. The instances would just hang for no reason and instead of wasting time troubleshooting, I’d rather focus on the code itself
- Wrestled with libvrt’s documentation and finally was able to collect CPU statistics using the API (see snippet below). The documentation sometimes says you need to pass in a struct when they mean pointer (and vice versa)
```
cpuTime: 119900000000
root@aos-kvm-ubuntu-18:/tmp# ./vcpu-scheduler 
Num domains: 2
Active Domain IDs:
2
1
State: 1, nrVirtCpu: 2, cpuTime=311380000000
Domain CPU Stats:
cpuTime: 162140000000
cpuTime: 144030000000
State: 1, nrVirtCpu: 2, cpuTime=268990000000
Domain CPU Stats:
cpuTime: 142960000000
cpuTime: 119900000000
```
Work
- Firmed up my design document at work, addressing other engineer’s feedback that was sprinkled in the Quip document
- E-mailed Asians@ so that they include my upcoming event in the newsletter
Organization
- Using my ScanSnap IX1500, I scanned all my loose documents (e.g. tax statements, W-2 copies, etc) and then shredded them, clearing up space on my desk and just overall getting a little more organization
Friends and Family

Elliott getting stuck in 3 tier shelf
- Popped into Wells Fargo Bank to transfer earnest money to the escrow company. I had originally tried calling the bank at around 04:00 AM (when their support center opened) to have both enable wire transfers on my account and bump up the transfer limit but quickly realized that sending tens of thousands of dollars should be probably be done in person at the branch
- Unexpectedly watched Elliott for the about 3 hours yesterday. Jess was already running behind for the inspection for our new home located in Renton and Elliott was still napping. So instead of waking her up, she left her with me in the middle of the work day and now I truly understand the plight of parents all around the world, parents who need to work from home while taking care of young children: it’s impossible.
- Scheduled follow up veterinarian appointments for both Metric and Mushroom. Metric’s ear developed a little fissure in her ear, looking as if a tick made a home and then evacuated. And on the back of mushroom’s neck, there’s a maybe 5 or 6 white head looking bumps that are located in the place where I had applied her flea medication a couple weeks ago
Today

Organization
- Call Well’s Fargo (again) and see if they can bump the limit on my wire transfer since I want to avoid going to a Wells Fargo branch due to COVID-19
- Migrate straggling sticky notes that are sitting on desk and process them into “Writing Ideas” or “Inbox” in Omnifocus
Graduate School
- Finish memory virtualization series
- Write a few more lines of C code to get a better sense of how I’m going to write the scheduler (all a bit fuzzy right now). Not sure how scheduler is going to integrate with KVM and not sure what algorithms I’ll select and implement
Work
- Meet with principle network engineer to close out open questions on my design document
- Attend weekly operations meeting and discuss events that popped up throughout the week
Family
- Carry out morning routine: walk with Elliott and Jess and the dogs, blend up a delicious strawberry and blueberry and banana smoothie (thanks to Jess, who picked up the necessary ingredients from Trader Joe’s yesterday), feed the dogs their raw food
- Reply to loan broker and see if they can lock in the 2.875% rate for our mortgage
- Hold a discussion with Shuk (our realtor) and finalize what issues that popped up during home inspection need to get fixed before we move forward into the next step
What are you grateful for?
Spending 1 on 1 time with little Elliott yesterday (despite needing to watch her last minute during a work day). I cannot really explain it but I think I’m developing some sort of paternal love for my daughter, feelings that I’ve never experienced before, not for any human and not for any of my beloved pets. I cannot really describe the emotion … but she’s able to put a wrinkle in my nose just by sitting there and shooting me a smile.
Feelings
After 3 hours of watching Elliott yesterday, I was wiped out, taxed physically and emotionally. That little 11 month year old crawls faster than a rattlesnake and I’m having a hard time keeping up with her. She puts everything in her mouth, from dog hair to shreds of paper (but I would like to call out a victory from yesterday, a proud dad moment, when I was able to free a piece of paper from her mouth before she swallowed it: dad goals).
September 4, 2020
Daily Review – Day ending in 2020/09/02
Writing
- Blogged and published an entry for my daily review
- Blogged and published an entry that shows the steps for building librty’s documentation from source code (since the manual no longer exists on the official website)
Mental and Physical Health
- Met with my therapist, who I see every week (except last week since I had to cancel the session due to being on call)
- Shared how I enjoy mentoring other people not just on technology but on the human element of our work. That’s the good stuff.
- He thinks that my preference for having an organized mind stems from survival techniques that I developed as a young child
Music
- Practiced singing major and minor scale from memory (no instrument leading me with any tones), a skill I picked up from my guitar instructor
Graduate School
- Generated the libvrt documentation from source code
- Installed libvrt-dev, enabling me to compile the first executable binary with -lvirt flags passed to gcc
- Compiled program using example source code from documentation that I built
- Watched second part of “memory virtualization” lecture and learned that shadow pages map pages from the guest virtual operating system to machine page number
Work
- Met with someone AWS Networking , the two of us chatting about the new feature that we are going to launch in Q1 2021
- Debugged a crash with the fuzzer (frustrating because I’m unable to reproduce the crash and neither can the Principle engineer on the team)
Friends and Family
- Bathed Elliott last night, the bath running longer than usual (30 minutes instead of 10)
- Feeling very pressured and nervous around moving to Renton, so much needed to get done to make it happen (e.g. pack all of our belongings, hire a moving company, cleaning up existing house that we rent). Need to continue taking deep breathes and chip away at each task, slowly, one by one
Today

Organization
- Call Well’s Fargo (again) and see if they can bump the limit on my wire transfer since I want to avoid going to a Wells Fargo branch due to COVID-19
- Migrate straggling sticky notes that are sitting on desk and process them into “Writing Ideas” or “Inbox” in Omnifocus
Graduate School
- Wrap up memory virtualization series
- Write a few more lines of C code to get a better sense of how I’m going to write the scheduler (all a bit fuzzy right now). Not sure how scheduler is going to integrate with KVM and not sure what algorithms I’ll select and implement
Work
- E-mail Asians@ so that they include my upcoming event in the newsletter
Family
- Carry out morning routine: walk with Elliott and Jess and the dogs, blend up a delicious strawberry and blueberry and banana smoothie (thanks to Jess, who picked up the necessary ingredients from Trader Joe’s yesterday), feed the dogs their raw food
- Schedule follow up veterinarian appointments for both Metric and Mushroom
What are you grateful for?

To be in a position where I can (and have been for the last 4 years) attend therapy, thanks to my insurance covering a large portion of the bill. Everyone should be able to afford health care.

Oatmeal breakfast that Jess whipped up

Also grateful for a delicious oatmeal breakfast (above) that Jess cooked for breakfast.

Feelings
- Same as yesterday: Simultaneously excited and nervous about buying and moving into a new home
September 3, 2020
Daily Review – Day ending in 2020/09/01
Yesterday

Writing
- Blogged and published an entry for my daily review
- Blogged and published an entry that snapshots my understanding before starting on project 1 of advanced operating systems: writing a scheduler and writing a memory coordinator
Music
- E-mailed the singing instructor that I’ve been seeing for the last couple years, informing her that lately I’ve been too busy and had to shift around my priorities, now that I’ve stepped into fatherhood. I sorely miss singing and felt that the activity brought a breathe of fresh air into my life. Maybe I can continue and maybe I can do one off lessons: that’s always an option.
Graduate School
- Starting working on project 1 by ensuring that I can launch the virtual machines inside of my virtual box environment. Ran into a slew of issues that I’ve document and will publish on this blog
Work
- Presented and my design document for a new feature/service that AWS will be offering in the future. I had to shake off my nervousness, a feeling I get despite how well prepared and despite how number of years I’ve practiced and polished my public speaking skills
- Starting debugging a crash discovered by our Fuzzer. I never dealt directly with the fuzzer so this is a great learning opportunity to not only fix a problem but deeper understand what the fuzzer exactly is doing
Friends and Family

Excited to design and decorate my new home office
- Bathed Elliott last night. She only lasted about 5 minutes (about 1/2 to 1/3 of the time we usually take a bathe for) since she was so sleepy, despite her clocking in a one and a half hour nap, an hour longer than her other naps. Maybe she’s going through some sort of growth spurt? Maybe she’s sleeping better because I hung up curtains in room that shield her from the setting sun?
- Video chatted with Martin, the two of us discussing software and architecture design for an authentication system he is working on. Nice that I can share my thoughts around trade offs, trade offs that I’ve picked up from both working at Amazon over the years and from graduate school. For example, talking about the trade offs of caching and caching is not free: need to tackle cache consistency and cache coherency.
- Panicked panicked panicked. The offer that we put in on the house the day before has been accepted and my wife and I are officially pending on a new house located in Renton. Although I’m nervous and scared and will miss North Seattle, I know that this relocation is the right step for our family. Elliott needs more space and seeing her crawl around the living room — over and over and over again — reaffirms my decision. Not only that, but I can finally build myself a real work from home office, one that feels warm and one that I can call my own.
Today

Organization
- Plan day and week out by reviewing OmniFocus forecast events
- Process e-mail inbox down to zero
- Migrate sticky notes (written down while walking dogs in the morning) into writing tracker and OmniFocus
Graduate School
- Begin second series of lectures for advanced operating systems, lectures on “Memory Virtualization” (exciting stuff, I think)
Work
- Revisit the open comments from design review and follow up with AWS Networking teams
Family
- Check work calendar and check if I can perform the home inspection at 2:00 PM on Thursday
What are you grateful for?

Despite the fact that we’re in the midst of a pandemic, despite that the massive layoffs in America and 10% unemployment rate, I’m fortunate enough to be in a position to have earned and saved enough money to buy a house. I feel both very blessed and also guilty at the same time. I acknowledge my hard work and perseverance but also acknowledge that I could not have done this on my own: so many people have helped me along the way in my life. I must continue to return the favor.

Feelings
- Simultaneously excited and nervous about buying and moving into a new home
September 2, 2020
Daily Review – Day ending in 2020/08/31
Yesterday

Most interesting part of yesterday was a 1 on 1 video Chime chat that my co-worker scheduled with me. What started off with me stepping through our code base (for a set of features I added to the dataplane) morphed into a lengthy discussion around mental health and burning out and the idea of feeling confident in our own abilities. They had shared with me that they were starting to feel physically ill and after paying visits to the doctor for a full body check up, they discovered their symptoms manifest from mental health issues.

I’m very familiar with these feelings and can empathize.

In turn, I shared that I prioritize mental health and well being above all else and that I attend weekly therapy sessions (for the past four years) to deal with very similar issues that they are encountering: learning how to set boundaries, learning how to believe in oneself (i.e. imposter syndrome). It’s not as if I’ve completely overcome and mastered these problems; in fact, I wrestle with these issues — every day — but I now have have tools and data points to support me.

Although I love working as a software engineer, I sometimes wonder if I would pivot into a position or line of work where I can directly help people. It feels meaningful.

Writing
- Kept up with my writing cadence, publishing two small blog posts (a daily review and summary notes for the SPIN operating system structure)
Music
- Sang and played my own original guitar song I titled “Elliott’s in the house” for little Elliott while she was bouncing up and down on the couch while dinner, her mom guiding spoon fulls of avocado and noodles in her dime sized mouth
Graduate School
- Read first paper “SPIN”, performing a first pass on the paper (good to reinforce the learning material since there’s so much information packed into the lectures)
- Finished watching lectures on “Introduction to Virtualization” (learned that with paravirtualization, less than 2% code modification is necessary to integrate nicely with the underlying hypervisor)
Work
- Represented my team at the weekly operations meeting for my organization, stepping through all the high severity issues that I encountered and how they affected our metrics
- Finished a first draft of my design document, a paper that I am presenting to my organization today
- Met with a colleague of mine and confirmed that he will be participating in my fireside chat that I am organizing on behalf of Asians@ Amazon (I’m on the professional and development committee)
Family
- Gathered all the necessary documents (e.g. 2 months of pay stubs, last 2 years of W-2, etc) to submit to our lender since we needed to send an offer in for a house by today and needed a preapproval letter.
Today

Organization
- Plan day and week out by reviewing OmniFocus forecast events
- Process e-mail inbox down to zero
- Migrate sticky notes (written down while walking dogs in the morning) into writing tracker and OmniFocus
Graduate School
- Begin second series of lectures for advanced operating systems, lectures on “Memory Virtualization” (exciting stuff, I think)
Work
- Polish one pager design document for work and present
- Review dashboard in preparation for organization wide, weekly operations meeting
Family
- Morning routine of walking dogs at local Northacres park, blending a strawberry and banana smoothie (wait: no much coconut milk so that’s out of the question), eating lunch together, bathing Elliott as part of her night time routine
- Put together loan documents so that we can get a pre-approval for house that we are putting an offer in
Word of the day

insolent – adjective (adj) – showing a rude and arrogant lack of respect.

My strict teacher does not tolerate insolent behavior in her classroom

What are you grateful for?

I’m glad I’m writing down what I’m grateful for because its so easy for me to get trapped inside my own brain, my own world, when work (on call specifically) feels like a tornado.
- Jess being an amazing mom (so patient, so communicative) and leading parenting my example
Feelings
- I had lamented and avoided writing a paper at work and realized, after setting ink on paper for just a few minutes, I actually enjoyed the task but had to overcome my own fear of “not knowing enough” or “looking stupid” in front of my colleagues
- As mentioned above in the summary, I feel good when I can help and serve others. Not in a purely intellectual way, but mentally and emotionally. This is important and perhaps will steer me in a new (or same or different) direction with my career
September 1, 2020
Daily Review – Day ending in 2020/08/30
Yesterday

Writing
- Wrote and published an article on Time Management Tip #1: Pomodoro Technique. How I use the technique and how I’ve adjusted it to fit my needs.
Music
- Had my bi-weekly virtual guitar lesson with Jared. This was the first lesson back, after he took a 1.5 month sabbatical, a vacation he deserves and a vacation that he takes once a year. During the lesson, we didn’t actually focus on any mechanics of guitar this time, which was fine by me because instead, we had a lengthy discussion around where we want to take guitar lessons: what’s the point and what do I want to work towards. More on this in a separate blog post.
Graduate School
- Submitted both homework assignments (apparently a day before it was due: so hooray me) and watched the video lectures while sitting in the back seat of the car, while my wife drove us from Seattle to Renton (see house hunting below, under family section).
Organization
- Miss: I had a scheduled call at 06:00 PM PST with a fellow class mate and I missed the chat. For two reasons: my phone died on the way home and two the event was not scheduled in my calendar, no alerts or notifications. Fortunately, he was still online at the time so we were able to meet over Google Hangouts despite me being 30 minutes late or so.
Family
- House hunted in Renton, viewing three homes and one of which we will submit an offer. We’ll go in at listing price and per our realtor, a friend of my sisters, escalate to about 20k above if there are competing offers. I could really see Jess and I living in the home, the square footage practically doubling the size of our current rental. Funny how having kids changes things. Up until a week or two ago, I opposed the idea of moving to the suburbs, opposed to moving away from Maple Leaf/North Seattle: all favorite local cafes are within walking distance, the amazing off leash dog parks like Magnuson, top notch vegan restaurants. But since COVID hit, my view has changed. Based off of the amount of time we spend at home, which is about 23 out of the 24 hours in the day, I’d love for Elliott to have a little more space, more than the 15 square feet of crawling space that we have now.
Today

Organization
- Plan day and week out by reviewing OmniFocus forecast events
- Process e-mail inbox down to zero
- Migrate sticky notes (written down while walking dogs in the morning) into writing tracker and OmniFocus
Graduate School
- Begin on lecture series for the week, the lecture focusing on virtualization (curious what this means)
- Write up notes taken over the weekend on L3 Microkernel, SPIN, and Mach, three different underlying OS structures, each with their own trade offs
Work
- Finish one pager design document (due tomorrow)
- Review dashboard in preparation for organization wide, weekly operations meeting
Family
- Morning routine of walking dogs at local Northacres park, blending a strawberry and banana smoothie (wait: no much coconut milk so that’s out of the question), eating lunch together, bathing Elliott as part of her night time routine
- Put together loan documents so that we can get a pre-approval for house that we are putting an offer in
Bonus Image

Elliott standing up at the beach

Word of the day

decry – adjective (adj) – publicly denounce

“SPIN and Exokernel used Mach as a basis for decrying Microkernel-based design”

What are you grateful for?

I’m glad I’m writing down what I’m grateful for because its so easy for me to get trapped inside my own brain, my own world, when work (on call specifically) feels like a tornado.
- To be in a financial position to even be looking to buy a home in the midst of COVID-19
Feelings
- Nervous about the move. Just like Jess. Leaving behind North Seattle is scary: we’ve grown to love the city. Will we regret it? Maybe. But the way I see it is that this decision (of moving to Renton) is not a one-way door: we can always move back and rent in North Seattle. Sure, we might not get the same awesome rental that we been living in for 4+ years but that’s a fair trade off. We are doing the right thing by listening to our gut and being heedful to our fears.
August 31, 2020
Daily Review – Day ending in 2020/08/27
What’s your chief aim for today

Learning that while I’m on call for my team, I very rarely hit my targets for the day (see section on “How did yesterday go”). Not only that, but I’m drained by the end of the day, little to no CPU cycles left for studying (although this window of time is ripe, since Elliott is asleep). Nonetheless, I’ll still call out what I want to accomplish today since these little tasks, when added together, help me fulfill my long term goals (a separate page for this will be pinned up on my blog soon).
- Writing
  - Write and publish one piece on this blog
- Graduate School
  - Complete typing up answers for assignment #1
  - Edit answers for pre-lab assignment
  - Watch and take notes on second module in advanced operating systems (Lesson 2)
- Organization
  - Review Inbox items sitting in OmniFocus
  - Scan projects sitting in my OmniFocus
- Work
  - Continue to chip away at writing one-pager design for a prototype for new feature at work
  - Survive today’s on call and flurry of operational issues that may arise
How did yesterday go?

Basically, when I’m call, the entire day is blurry, the day flashing by right before my eyes. During the day, I’m in a completely different state, a reactive mode, dealing with whatever gets thrown at me from auto cut tickets to Chime messages or Slack messages or e-mails. Chaotic days like this put me in a despondent state.
- Another emotionally draining day, thanks to the on call operations for the week. I got paged at 12:00 AM and after troubleshooting for about half an hour, I was unable to fall back asleep for at least another 45 minutes. When getting woken up in the middle of the night, I find that there’s a very small window — about 15 minutes — in which I can fall back asleep. After that amount of time has elapsed, falling back asleep requires serious deep breathing
- Because of the interrupted sleep, I was unable to wake up early and use that morning slot to squeeze in studying before work. I’ll borrow back some time during the day to make up for it though. Oh yea, I also am a bit flustered that I was unable to comfortably hang out out with my daughter and wife and dogs because I did end up getting paged this morning (at around 07:00 am). Fortunately, we had already walked at the park so I was nearby my laptop: so no breach in service level agreement (SLA).
- Did not get to put any study time in (as mentioned above) so I’ll need to carve out time today to catch up with watching videos lectures and typing up my homework assignment for advanced operating systems.
Word of the day

remiss – adjective (adj) – lacking care or attention to duty; negligent.

“it would be very remiss of me not to pass on that information”

What are you grateful for?

I’m glad I’m writing down what I’m grateful for because its so easy for me to get trapped inside my own brain, my own world, when work (on call specifically) feels like a tornado.
- My German Shepherd dog (Metric) is in good health (she eats raw, gets daily exercise) and behaves wonderfully with my daughter. I do feel a pang in my heart when I notice small signs that she’s aging, signs like an additional grey whisker sprouting out.
- Jess ordering two delicious vegan sandwiches for lunch, sandwiches glazed with moist hummus and filled with soft chick peas. Yummy.
Feelings
- As mentioned above, on call operations for my team (and many other teams in Amazon) can drain every ounce of energy. Every time my phone sounds an alarm, my body releases cortisol and stress washes over me. Granted, there are some weeks where very little issues crop up: but that’s the not the norm as of late. But is this how I want to live my life 24×7, once every 5-6 weeks? Sure, the work is rewarding and challenging and intellectually stimulating but one entire week of pure exhaustion may not be worth the cost
- Seeing my wife watch over both my daughter and niece (Maiya) sends butterflies to my stomach. She’s patient and gentle with them, two important qualities for anyone.
References
1. https://www.theatlantic.com/culture/archive/2020/08/lovecraft-country/615259/
August 28, 2020
Daily Review – Day ending in 2020/08/26
What’s your chief aim for today
- Writing
  - Write and publish one piece on this blog
- Graduate School
  - Complete typing up answers for assignment #1
  - Edit answers for pre-lab assignment
  - Watch and take notes on second module in advanced operating systems (Lesson 2)
- Organization
  - Review Inbox items sitting in OmniFocus
  - Quickly glance over project
- Work
  - Continue to chip away at writing one-pager design for a prototype for new feature at work
  - Survive today’s on call and flurry of operational issues that may arise
How did yesterday go?
- Realized that checking my work laptop first thing in the morning for just that one little thing doesn’t work: I get sucked in and begin tackling other items, including checking my e-mail inbox, or slack messages, or whatever. So my rule of thumb is to not flip open the lid of my work laptop until I am ready to fully work
- Work drained me. All the back to back operational issues not only enervated me but derailed my entire day, something I should just expect since on call is so variable
- Being on call yesterday, I was unable to break away from the laptop while my mom was visiting Seattle, not getting to spend much quality time with her. Bummer. These feelings make me reevaluate work (see Feelings section)
Word of the day

underscore – verb (n) – to underline or to emphasize

Even Atlanta, Donald Glover’s Twin Peaks–esque FX series, toys with the supernatural and the surreal to underscore points about race made in its quieter moment¹.

In an internal memo, Stan Deal, chief executive of Boeing Commercial Airplanes and Greg Hyslop, Boeing’s chief engineer, sought to underscore the message that safety is the company’s top priority ²

What are you grateful for?
- My sister delivering a healthy little girl, her birthing free from any major complications (unless you consider an epidural a complication)
Feelings
- As mentioned above, on call operations for my team (and many other teams in Amazon) can drain every ounce of energy. Every time my phone sounds an alarm, my body releases cortisol and stress washes over me. Granted, there are some weeks where very little issues crop up: but that’s the not the norm as of late. But is this how I want to live my life 24×7, once every 5-6 weeks? Sure, the work is rewarding and challenging and intellectually stimulating but one entire week of pure exhaustion may not be worth the cost
- Seeing my wife watch over both my daughter and niece (Maiya) sends butterflies to my stomach. She’s patient and gentle with them, two important qualities for anyone.
References
1. https://www.theatlantic.com/culture/archive/2020/08/lovecraft-country/615259/
August 27, 2020