Author: mattchung

  • Python progression path – quiz

    Python progression path – From apprentice to guru is a popular StackOverflow post. To categorize whether a person should take his beginner/intermediate course, one of the commentors posted this question:

    python progression quick

    I can better answer this question after reading Fluent Python. Example 1 and Example 2 deal with immutable and mutable data types – respectively. Let’s address each example individually.

    Example 1

    The id built in method returns the object’s memory address. We’ll use this to inspect the memory address for x and ythroughout the examples.

    x and y point to the same memory address when they are first assigned.

    >>> x = 42
    >>> y = x
    >>> id(x)
    4298172632
    >>> id(y)
    4298172632
    

    Because the value of x is immutable (e.g INT) and cannot be modified, by definition, a new memory address is allocated when x is modified. But, y‘s memory remains unmodified:

    >>> x = x + 1
    >>> id(x)
    4298172608
    >>> id(y)
    4298172632
    >>>
    

    Example 2

    Like Example 1, the memory address starts off the same:

    >>> x = [1,2,3]
    >>> y = x
    >>> id(x)
    4299948256
    >>> id(y)
    4299948256
    

    Now that we are dealing with mutable data types, x can be modified in place and the memory address does not change:

    >>> id(x)
    4299948256
    >>> id(y)
    4299948256
    

    Mutable default arguments

    This quiz seems trivial but it isn’t. Understanding this will prevent a novice mistake of using mutable default arguments.

    >>> def update(x, y=[]):
    ...     y.append(x)
    ...     return y
    ...
    >>> list1 = ['a', 'b', 'c']
    >>> list2 = [1, 2, 3]
    >>> update('d', list1)
    ['a', 'b', 'c', 'd']
    >>> update(4, list2)
    [1, 2, 3, 4]
    >>> list1
    ['a', 'b', 'c', 'd']
    >>> list2
    [1, 2, 3, 4]
    >>> update('a')
    ['a']
    >>> list3 = update('b')
    >>> list3
    ['a', 'b']
    >>> list4 = update('hello')
    >>> list4
    ['a', 'b', 'hello']
  • Relearning how to juggle

    I learned how to juggle 15 years ago. I remember relentlessly practicing in my room late into the night. Within a week, I was comfortably juggling three balls.

    Recently, I stumbled across YouTube video of someone juggling four balls and was very impressed. How much more difficult is juggling four balls than three, I thought to myself.

    So, I watched this YouTube tutorial and his pre-requisites to juggling four balls are:
    • Juggle three balls for 1 minute
    • Juggle two balls in each hand for 30 seconds

    In my mind, juggling two balls felt like a step backwards. So, I dismissed his prerequisites.

    I picked up four tennis balls and after 15 minutes of getting nowhere, I grew increasingly frustrated. My years of juggling three balls did not, in fact, make juggling four balls any easier. According to this video, it takes this much time to be able to juggle. Here’s what should be expected:

    • 3 Balls – Days to Weeks
    • 4 Balls – Weeks to Months
    • 5 Balls – Months to Years

    I decided to stop juggling four balls and go back to the fundamentals: two and three ball exercises. I’m glad I did. After spending the first week “starting over”, here’s my progress:

    early attempt of juggling four balls

    In a way, glad that juggling four balls didn’t come easy. Sometimes, you have to go back a few steps to move forward.

  • Belgium terrorist attack and the media

    Belgian was this morning. I send my thoughts and prayers to those in Belgium, but I’m worried about the media’s knee jerk accusations of Muslim terrorists. At the moment, there’s no concrete evidence. But why is the live feed incessantly hinting at ISIS and Muslims?

    I can’t stay updated without feeling the media is inculcating anti muslim propagnda.

    I question the media’s “experts”. The media ensures that expert testimtony aligns with its agenda. Sky News interviewedAnne Speckhard and she immediately concluded that the incident must be related to Salah Abdelslam. I’m not suggesting that it isn’t, but its too quick to point the finger without facts.

    “It’s probably the group that Salah Abdelslam is part of …. so I think they just accelerated their plans … so they wouldn’t get rounded up”, said Anne Speckhard.

    “So you think this is related to the arrest of Salah Abdelslam on Friday and that perhaps that the planned attacks have been brought forward as a result of that?”

    “Absolutely.”

    “One gunshot afterwards and there was one man speaking Arabic afterwords. And then I heard a boom, an enormous explosion”

    I’m disappointed.

  • AWS Lambda part 2 – packaging and deploying

    I received positive feedback on my AWS Lambda presentation in London. This post discusses how to package and deploy your lambda function.

    I’m sure there are other ways but I wanted something simple. This allows me to maintain separate enviroments (i.e “dev”, “production”).

    Makefile

    I use Make to create a package (i.e “sample-app.zip”). The dependencies are installed into a virtualenv which are appended to the package.

    Create the function

    make build-dev
    make create-dev
    

    Deploy the package

    make build-dev
    make update-dev
    
    PROJECT = sample-python-app
    FUNCTION = $(PROJECT)
    REGION = us-east-1
    .phony: clean
    clean:
    rm -f -r $(FUNCTION)*
    rm -f -r site-packages
    build-dev: clean
    aws s3 cp s3://$(FUNCTION)/settings-dev.yml settings.yml
    zip -r $(FUNCTION)-dev.zip . -x *.git* tests/*
    mkdir -p site-packages
    virtualenv $(FUNCTION)-dev
    . $(FUNCTION)-dev/bin/activate
    pip install -r requirements.txt
    cd site-packages; cp -r $$VIRTUAL_ENV/lib/python2.7/site-packages/ ./
    cd site-packages; zip -g -r ../$(FUNCTION)-dev.zip .
    create-dev:
    aws lambda create-function \
    –handler main.lambda_handler \
    –function-name $(FUNCTION)-dev \
    –region $(REGION) \
    –zip-file fileb://$(FUNCTION)-dev.zip \
    –role arn:aws:iam::XXXX:role/$(FUNCTION)-dev \
    –runtime python2.7 \
    –timeout 120 \
    –memory-size 512 \
    update-dev:
    aws lambda update-function-code \
    –function-name $(FUNCTION)-dev \
    –zip-file fileb://$(FUNCTION)-dev.zip \
    –publish \
    delete-dev:
    aws lambda delete-function –function-name $(FUNCTION)-dev

    Lambda deployment package

    Make sure you include all the dependencies in the package. You need to copy the contents of site-packages, NOT the directory itself. If you copy site-packages itself, lambda will not be able to find the modules.

    The contents will look something similar to:

    main.py
    netstorage/
    netstorage/__init__.py
    requests/
    requests/__init__.py

    Testing locally

    Instead of constantly deploying the function, I wanted to test my changes locally.

    When creating an AWS lambda function, you need to specify the handler gets executes. In the gist above, I specify it with –handler main.lambda_handler. So testing locally is simple as:

    if __name__ == "__main__":
        lambda_handler(None, None)
  • Monitoring background processes with SumoLogic

    This post discusses one way we monitor our background process – which is different than how we monitor our web services. It’s difficult when you can’t send a request/poll the service. I wanted something more than checking if the process is alive. One solution we came up with is is using syslog with a log aggregration service.

    Most log aggregration platforms (i.e “Splunk, Log Entries, Sumologic,”) can send an alert if some string (or regex) is found in the the logs. This is pretty common; what about alerting when logs are NOT found?

    Basic Setup

    Our process is managed by supervisor and it logs to stdout. Supervisor logging plugin sends logs to our centralized syslog collector, which forwards the logs to Sumologic. We then configure alerts based off of search results.

    Configuring the alert

    First, create your search (here’s a cheatsheet). Next, configure it:

    • Library -> Edit Search
    • Scheduled this Search
    • Number of results Equal to = 0

    Screenshot:

    Sumologic scheduled search equal to zero/0That’s it. You can integrate this with your Pager Duty account or just have it send directly to an e-mail address.

  • Getting older

    I was standing outside the car, wiping the dogs’s feet when I overheard a voice.

    “Are you a young lady or a man?”

    Did I hear him right? I continued wiping my dogs’s feet and responded:

    “Well, HER name is Metric. And SHE’S a German Shepherd.” I wanted to make it clear that that we were talking about the dog.

    “NO”, he responded confidently. “I’m asking YOU if you are a BOY or GIRL.”

    I turned to Jessica, my fiancée, who was sitting in the car, laughing uncontrollably. I started laughing too.

    You see, I suddenly realized that the the old man was confused by my long hair and it didn’t help that my back was turned to him.

    I couldn’t get mad at him. He spoke his mind – something I fervently believe in.

    Zero Inhibition

    Old people speak their minds. It’s the same for kids. In Master of None, there’s a scene where two kids enter an ice cream shop and start pointing people out by their skin color:

    “Black. White. Yellow. Black. Black.”

    Can you imagine an adult saying that (only person I can think of is my Viet Grandma).

    I am constantly trying to find the balance of thinking twice before saying something and being assertive. It’s an ongoing battle.

  • Putting your mentor on a pedestal

    Presenting at DevOps London ExchangeLast night, I presented (deckslide here) on AWS Lambda at DevOps London Exchange. I really enjoy public speaking, but it wasn’t always that way.

    In fact, I used to hate it – feared it.

    I vividly remember an embarassing instance in high school Spanish. My classmate and I had to do a presentation. I got up, and stood in front of the class for 20 minutes.

    That’s it. I just stood there. I didn’t say a SINGLE word the entire time.

    Fast forward to college. By this time, I really needed help with public speaking. My uncle suggested joining Toastmasters.

    I visited a few clubs and I wasn’t very impressed. But then, I discovered Sherman Oaks Toastmasters.

    I met Peter Bunce. He was very passionate about Toastmasters. He was very discplined. For 20 years, he rarely (if ever) missed the weekly meetings. His entire life was devoted developing the Toastmasters club.

    We used to get coffee and dinner. He assigned himself as my mentor. Honestly, I was hoping for someone else. He was tooeccentric. He didn’t encapsulate all the characteristics of my ideal rolemodel.

    Setting the right expectations

    I’ve always had unreal expectations from those who I looked up to. Peter was no exception. Him being a great speaker wasn’t enough for me. I had unrealistic expectations of what one mentor can offer – let alone any person.

    “If you should end up with a teacher who doesn’t seem right for you, first look inside.”

    —George Leonard

    A note to my mentor

    I sent Peter a message on Facebook this morning. But he’ll never read it.

    Peter Bunce passed away a few weeks ago. I wish that I had reached out to him sooner.

    If there’s someone who mentored you, be it informally or formally, or touched your life in some way, don’t wait to tell them that they moved you. Take a moment to reach out to them, to say thank you.

  • SpeedCurve Library: Travis CI + coverage + tox

    I’m writing a python library (speedcurve.py). At work, we’re using SpeedCurve to track our web performance. SpeedCurve’schangelog reveals can trigger a deployment (and a series of tests). I’m replacing our curl+ansible commands with this. I plan on integrating this with slack as well.

    This project is heavily influenced by github3.py. I’ve been contributing frequently (plan on continuing to) and find it elegant. You’ll notice A LOT of similarities.

    For this project, I want the following up front:
    • Proper Documentation
    • 100% test coverage and flake8 checks
    • Integration with Travis CI

    This post focuses on configuring tox to execute coverage.

    Configure tox + coverage

    Since tox does NOT play nicely with pipes (|), this simple shell scripts checks for 100% test coverage:

    #!/bin/sh
    
    coverage run --source speedcurve -m py.test
    coverage report | tail -n 1 | grep '^TOTAL.*100%$'
    

    This goes in tox.ini

    [testenv:coverage]
    commands =
        ./coverage-check.sh
    
  • Implementing Licenses API for github3.py

    I came across this license issue while searching on GitHub. So, I thought I’d give it a shot. I pinged sigmavirus24 in #github3.py seeing if this was a feature I could implement. He gave the thumbs up and I was off.

    Testing manually first

    Before any implementation, I always like to get a better understanding of the API by using good ol’ fashion curl.

    curl https://api.github.com/licenses \
    -H "Accept: application/vnd.github.drax-preview+json"
    

    Sending a custom Accept: Header

    In the code base, most classes inherit from github3.GitHubCore, which inherits from requests.Session. We can pass in**kwargs, and requests.Session.get accepts a headers as a kwarg. So, we can pass a custom Accept: header like so:

    headers = {
        'Accepts': 'application/vnd.github.drax-preview+json'
    }
    
    url = self._build_url('license')
    json = self._json(self._get(url, headers=headers))
    

    How to add attributes to License model

    By default, the model will not expose any attributes. How do we do that? The key is implementing __update_attributesmethod.

    github3.licenses.License inherits from github3.models.GitHubObject, which calls __update_attriubtes in its__init__.

    class GitHubObject(object):
        """The :class:`GitHubObject <GitHubObject>` object. A basic class to be
        subclassed by GitHubCore and other classes that would otherwise subclass
        object."""
        def __init__(self, json):
            super(GitHubObject, self).__init__()
            if json is not None:
                self.etag = json.pop('ETag', None)
                self.last_modified = json.pop('Last-Modified', None)
                self._uniq = json.get('url', None)
            self._json_data = json
            self._update_attributes(json)
    

    So, let’s add License attributes

    def _update_attributes(self, license):
        self.name = license.get('name')
        self.permitted = license.get('permitted')
        self.category = license.get('category')
        self.forbidden = license.get('forbidden')
        self.featured = license.get('featured')
        self.html_url = license.get('html_url')
        self.body = license.get('body')
        self.key = license.get('key')
        self.description = license.get('description')
        self.implementation = license.get('implementation')
        self.required = license.get('required')
    

    Writing test

    This guide is a great place to start. But, just a few pointers. For unit tests, copy/paste example data the API docs. For example, grab the JSON data from the license documentation. Save it under tests/unit/.

    For integration tests, you’ll need to perform HTTP request(s). The betamax wrapper will record it to tests/cassettes.

    Summary

    This feature was merged in this pull request. I really enjoy contributing to this project. Primarily since sigmavirus24 is a pleasure to work with and extremely helpful. He’s super patient and I appreciate he takes the time onboarding new contributors.

  • Contributing to github3.py

    sigmavirus24 tweetI’ve always been scared of open sourcing, despite wanting to get involved for a long time? Why?

    For a long time, I’ve programmed in isolation. But, I did enjoy it. Unfortunately, this leaves little (to none) opportunity for feedback and critism. Afraid of rejection? Afraid of not appearing as smart as I think I am ?

    Serepdentiosuly, I came across this post on reddit post. His comment looked warm and welcoming. Let’s give it a shot again.

    I received an e-mail from @sigmavirus24. Looks like he could use some help moving existings tests under tests/* to tests/unit and tests/integration. This is a great way to get familiar with the code base. I’m game.

    @sigmavirus24 posted this tweet. I was so happy and I hugged Jessica, who was sitting next to me when I saw it pop up on my feed. I didn’t anticipate it and it is what makes me enjoy working on F/OSS.

    I Was on IRC and dropped a note to @sigmavirus24 about implementing this new API feature. I think I’ll tackle this in parallel with migrating tests over.