Author: mattchung

  • Belgium terrorist attack and the media

    Belgian was this morning. I send my thoughts and prayers to those in Belgium, but I’m worried about the media’s knee jerk accusations of Muslim terrorists. At the moment, there’s no concrete evidence. But why is the live feed incessantly hinting at ISIS and Muslims?

    I can’t stay updated without feeling the media is inculcating anti muslim propagnda.

    I question the media’s “experts”. The media ensures that expert testimtony aligns with its agenda. Sky News interviewedAnne Speckhard and she immediately concluded that the incident must be related to Salah Abdelslam. I’m not suggesting that it isn’t, but its too quick to point the finger without facts.

    “It’s probably the group that Salah Abdelslam is part of …. so I think they just accelerated their plans … so they wouldn’t get rounded up”, said Anne Speckhard.

    “So you think this is related to the arrest of Salah Abdelslam on Friday and that perhaps that the planned attacks have been brought forward as a result of that?”

    “Absolutely.”

    “One gunshot afterwards and there was one man speaking Arabic afterwords. And then I heard a boom, an enormous explosion”

    I’m disappointed.

  • AWS Lambda part 2 – packaging and deploying

    I received positive feedback on my AWS Lambda presentation in London. This post discusses how to package and deploy your lambda function.

    I’m sure there are other ways but I wanted something simple. This allows me to maintain separate enviroments (i.e “dev”, “production”).

    Makefile

    I use Make to create a package (i.e “sample-app.zip”). The dependencies are installed into a virtualenv which are appended to the package.

    Create the function

    make build-dev
    make create-dev
    

    Deploy the package

    make build-dev
    make update-dev
    
    PROJECT = sample-python-app
    FUNCTION = $(PROJECT)
    REGION = us-east-1
    .phony: clean
    clean:
    rm -f -r $(FUNCTION)*
    rm -f -r site-packages
    build-dev: clean
    aws s3 cp s3://$(FUNCTION)/settings-dev.yml settings.yml
    zip -r $(FUNCTION)-dev.zip . -x *.git* tests/*
    mkdir -p site-packages
    virtualenv $(FUNCTION)-dev
    . $(FUNCTION)-dev/bin/activate
    pip install -r requirements.txt
    cd site-packages; cp -r $$VIRTUAL_ENV/lib/python2.7/site-packages/ ./
    cd site-packages; zip -g -r ../$(FUNCTION)-dev.zip .
    create-dev:
    aws lambda create-function \
    –handler main.lambda_handler \
    –function-name $(FUNCTION)-dev \
    –region $(REGION) \
    –zip-file fileb://$(FUNCTION)-dev.zip \
    –role arn:aws:iam::XXXX:role/$(FUNCTION)-dev \
    –runtime python2.7 \
    –timeout 120 \
    –memory-size 512 \
    update-dev:
    aws lambda update-function-code \
    –function-name $(FUNCTION)-dev \
    –zip-file fileb://$(FUNCTION)-dev.zip \
    –publish \
    delete-dev:
    aws lambda delete-function –function-name $(FUNCTION)-dev

    Lambda deployment package

    Make sure you include all the dependencies in the package. You need to copy the contents of site-packages, NOT the directory itself. If you copy site-packages itself, lambda will not be able to find the modules.

    The contents will look something similar to:

    main.py
    netstorage/
    netstorage/__init__.py
    requests/
    requests/__init__.py

    Testing locally

    Instead of constantly deploying the function, I wanted to test my changes locally.

    When creating an AWS lambda function, you need to specify the handler gets executes. In the gist above, I specify it with –handler main.lambda_handler. So testing locally is simple as:

    if __name__ == "__main__":
        lambda_handler(None, None)
  • Monitoring background processes with SumoLogic

    This post discusses one way we monitor our background process – which is different than how we monitor our web services. It’s difficult when you can’t send a request/poll the service. I wanted something more than checking if the process is alive. One solution we came up with is is using syslog with a log aggregration service.

    Most log aggregration platforms (i.e “Splunk, Log Entries, Sumologic,”) can send an alert if some string (or regex) is found in the the logs. This is pretty common; what about alerting when logs are NOT found?

    Basic Setup

    Our process is managed by supervisor and it logs to stdout. Supervisor logging plugin sends logs to our centralized syslog collector, which forwards the logs to Sumologic. We then configure alerts based off of search results.

    Configuring the alert

    First, create your search (here’s a cheatsheet). Next, configure it:

    • Library -> Edit Search
    • Scheduled this Search
    • Number of results Equal to = 0

    Screenshot:

    Sumologic scheduled search equal to zero/0That’s it. You can integrate this with your Pager Duty account or just have it send directly to an e-mail address.

  • Getting older

    I was standing outside the car, wiping the dogs’s feet when I overheard a voice.

    “Are you a young lady or a man?”

    Did I hear him right? I continued wiping my dogs’s feet and responded:

    “Well, HER name is Metric. And SHE’S a German Shepherd.” I wanted to make it clear that that we were talking about the dog.

    “NO”, he responded confidently. “I’m asking YOU if you are a BOY or GIRL.”

    I turned to Jessica, my fiancée, who was sitting in the car, laughing uncontrollably. I started laughing too.

    You see, I suddenly realized that the the old man was confused by my long hair and it didn’t help that my back was turned to him.

    I couldn’t get mad at him. He spoke his mind – something I fervently believe in.

    Zero Inhibition

    Old people speak their minds. It’s the same for kids. In Master of None, there’s a scene where two kids enter an ice cream shop and start pointing people out by their skin color:

    “Black. White. Yellow. Black. Black.”

    Can you imagine an adult saying that (only person I can think of is my Viet Grandma).

    I am constantly trying to find the balance of thinking twice before saying something and being assertive. It’s an ongoing battle.

  • Putting your mentor on a pedestal

    Presenting at DevOps London ExchangeLast night, I presented (deckslide here) on AWS Lambda at DevOps London Exchange. I really enjoy public speaking, but it wasn’t always that way.

    In fact, I used to hate it – feared it.

    I vividly remember an embarassing instance in high school Spanish. My classmate and I had to do a presentation. I got up, and stood in front of the class for 20 minutes.

    That’s it. I just stood there. I didn’t say a SINGLE word the entire time.

    Fast forward to college. By this time, I really needed help with public speaking. My uncle suggested joining Toastmasters.

    I visited a few clubs and I wasn’t very impressed. But then, I discovered Sherman Oaks Toastmasters.

    I met Peter Bunce. He was very passionate about Toastmasters. He was very discplined. For 20 years, he rarely (if ever) missed the weekly meetings. His entire life was devoted developing the Toastmasters club.

    We used to get coffee and dinner. He assigned himself as my mentor. Honestly, I was hoping for someone else. He was tooeccentric. He didn’t encapsulate all the characteristics of my ideal rolemodel.

    Setting the right expectations

    I’ve always had unreal expectations from those who I looked up to. Peter was no exception. Him being a great speaker wasn’t enough for me. I had unrealistic expectations of what one mentor can offer – let alone any person.

    “If you should end up with a teacher who doesn’t seem right for you, first look inside.”

    —George Leonard

    A note to my mentor

    I sent Peter a message on Facebook this morning. But he’ll never read it.

    Peter Bunce passed away a few weeks ago. I wish that I had reached out to him sooner.

    If there’s someone who mentored you, be it informally or formally, or touched your life in some way, don’t wait to tell them that they moved you. Take a moment to reach out to them, to say thank you.

  • SpeedCurve Library: Travis CI + coverage + tox

    I’m writing a python library (speedcurve.py). At work, we’re using SpeedCurve to track our web performance. SpeedCurve’schangelog reveals can trigger a deployment (and a series of tests). I’m replacing our curl+ansible commands with this. I plan on integrating this with slack as well.

    This project is heavily influenced by github3.py. I’ve been contributing frequently (plan on continuing to) and find it elegant. You’ll notice A LOT of similarities.

    For this project, I want the following up front:
    • Proper Documentation
    • 100% test coverage and flake8 checks
    • Integration with Travis CI

    This post focuses on configuring tox to execute coverage.

    Configure tox + coverage

    Since tox does NOT play nicely with pipes (|), this simple shell scripts checks for 100% test coverage:

    #!/bin/sh
    
    coverage run --source speedcurve -m py.test
    coverage report | tail -n 1 | grep '^TOTAL.*100%$'
    

    This goes in tox.ini

    [testenv:coverage]
    commands =
        ./coverage-check.sh
    
  • Implementing Licenses API for github3.py

    I came across this license issue while searching on GitHub. So, I thought I’d give it a shot. I pinged sigmavirus24 in #github3.py seeing if this was a feature I could implement. He gave the thumbs up and I was off.

    Testing manually first

    Before any implementation, I always like to get a better understanding of the API by using good ol’ fashion curl.

    curl https://api.github.com/licenses \
    -H "Accept: application/vnd.github.drax-preview+json"
    

    Sending a custom Accept: Header

    In the code base, most classes inherit from github3.GitHubCore, which inherits from requests.Session. We can pass in**kwargs, and requests.Session.get accepts a headers as a kwarg. So, we can pass a custom Accept: header like so:

    headers = {
        'Accepts': 'application/vnd.github.drax-preview+json'
    }
    
    url = self._build_url('license')
    json = self._json(self._get(url, headers=headers))
    

    How to add attributes to License model

    By default, the model will not expose any attributes. How do we do that? The key is implementing __update_attributesmethod.

    github3.licenses.License inherits from github3.models.GitHubObject, which calls __update_attriubtes in its__init__.

    class GitHubObject(object):
        """The :class:`GitHubObject <GitHubObject>` object. A basic class to be
        subclassed by GitHubCore and other classes that would otherwise subclass
        object."""
        def __init__(self, json):
            super(GitHubObject, self).__init__()
            if json is not None:
                self.etag = json.pop('ETag', None)
                self.last_modified = json.pop('Last-Modified', None)
                self._uniq = json.get('url', None)
            self._json_data = json
            self._update_attributes(json)
    

    So, let’s add License attributes

    def _update_attributes(self, license):
        self.name = license.get('name')
        self.permitted = license.get('permitted')
        self.category = license.get('category')
        self.forbidden = license.get('forbidden')
        self.featured = license.get('featured')
        self.html_url = license.get('html_url')
        self.body = license.get('body')
        self.key = license.get('key')
        self.description = license.get('description')
        self.implementation = license.get('implementation')
        self.required = license.get('required')
    

    Writing test

    This guide is a great place to start. But, just a few pointers. For unit tests, copy/paste example data the API docs. For example, grab the JSON data from the license documentation. Save it under tests/unit/.

    For integration tests, you’ll need to perform HTTP request(s). The betamax wrapper will record it to tests/cassettes.

    Summary

    This feature was merged in this pull request. I really enjoy contributing to this project. Primarily since sigmavirus24 is a pleasure to work with and extremely helpful. He’s super patient and I appreciate he takes the time onboarding new contributors.

  • Contributing to github3.py

    sigmavirus24 tweetI’ve always been scared of open sourcing, despite wanting to get involved for a long time? Why?

    For a long time, I’ve programmed in isolation. But, I did enjoy it. Unfortunately, this leaves little (to none) opportunity for feedback and critism. Afraid of rejection? Afraid of not appearing as smart as I think I am ?

    Serepdentiosuly, I came across this post on reddit post. His comment looked warm and welcoming. Let’s give it a shot again.

    I received an e-mail from @sigmavirus24. Looks like he could use some help moving existings tests under tests/* to tests/unit and tests/integration. This is a great way to get familiar with the code base. I’m game.

    @sigmavirus24 posted this tweet. I was so happy and I hugged Jessica, who was sitting next to me when I saw it pop up on my feed. I didn’t anticipate it and it is what makes me enjoy working on F/OSS.

    I Was on IRC and dropped a note to @sigmavirus24 about implementing this new API feature. I think I’ll tackle this in parallel with migrating tests over.

  • Mocking boto exception

    I was getting so frustrated.

    I knew how to raise the exception with side_effect=. But, how do you mock the exception?

    try:
        connection = connect_to_sqs
    except BotoServerError as m:
        if m.error_code == "AlreadyExistsException"
    

    To get it to work, I inherited the exception class – BotoServerError

    from boto.exceptions import BotoServerError
    
    class MockBotoServerError(BotoServerError):
        def __init__(self, error_code)
            self.error_code = error_code
    
    @mock.patch('cloudwhale.build.connect_to_sqs', side_effect=MockBotoServerError())
    

    Order of decorator

    Also, pay attention to the order of parameters. My assertions were failing left and right.