Thursday, 18 January 2018

Rapid Prototyping in Clojure with boot-clj

I find boot-clj to be great in doing rapid prototypes. It can be considered analogous to GroovyConsole. We can dynamically add dependencies, write, modify code, run, experiment all within a single file without having to create a project as with default lein setup. Create a folder for experiments and add boot.properties in it.
#https://github.com/boot-clj/boot
BOOT_CLOJURE_NAME=org.clojure/clojure
BOOT_VERSION=2.7.1
BOOT_CLOJURE_VERSION=1.8.0
Then we can create out prototype files, say pilot.clj with the below example template.
#!/usr/bin/env boot

;; To add some repository
(merge-env! :repositories [["clojars" {:url "https://clojars.org/repo/" :snapshots true}]])

(defn deps [new-deps]
  "Add dependencies to the namespace."
  (merge-env! :dependencies new-deps))

;; Add clojure
(deps '[[org.clojure/clojure "1.8.0"]])

;; Require
(require '[clojure.string :as str])
;; Import
(import '[javax.crypto.spec SecretKeySpec])

(println (str/upper-case "hi"))  ;; HI 

Thursday, 28 December 2017

Erlang DBX - An Idea Lost in Oblivion

When can I see BEAM (Erlang VM) specification book is what I am thinking right now. Currently we have Java Virtual Machine Specification and Java Language Specification for Java 8 books available. Having a nice book on the internals of the BEAM would be great.

BEAM is a register based VM and Hotspot is a stack based VM. Erlang code is converted into Core Erlang and then to BEAM bytecode by the Erlang compiler. I had this idea of implementing BEAM and the ERTS on a circuit as in Jezelle DBX for ARM which does Java direct bytecode execution in the hardware. It was mainly used during the Java ME era. I don't think this is used much these days.

My idea (brainstorming) was to have the entire ERTS on a chip. So the board can be considered analogous to Intel 8086 chip which has a CPU, memory unit, bus, bridges and all that mechanism. To have ERTS, it should have similar system. There will a CPU which executes the BEAM bytecode as opposed to x86 assembly instruction. Then we need to think about instruction sets timing, clock speeds. The registers required. Having the ERTS entirely on a physical circuit would make things more rigid as in how do we update things. So how about having parts in microcode and have a translation layer? The chip should be energy efficient, should outperform existing hardware, else we will have another Lisp Machine. We would require some sort of network on the chip if we are to have multiple Erlang VM on the same chip, which can communicate with each other. Should we think about security aspects of VM communication? What comes in mind is the QNX's Qnet protocol which allows the QNX Neutrino micro-kernel to transparently distribute and communicate over trusted devices.

Thinking about scheduler and job scheduling, BEAM has a master scheduler and they keeps changing. So should the CPUs be symmetric or is it better to have an asymmetric cores as in Cell BE microprocessor? If we use CISC, then the circuitry is going to be complex which requires more energy and produces more heat. So RISC then? Then we have the whole can of worms that modern CPUs comes with like out-of-bound execution, caching, pipelining, branch prediction and I have no much idea how to approach that in case of ERTS. Mostly we have to come up with our on mechanism as these are not part of the ERTS. The BEAM can have at most two copies of a module at the same time. But that does not count as caching.

From a tooling perspective, I know I can use VHDL and Simulink, OrCAD etc., but since I am not really a hardware oriented person and due to the lack of time and change in priorities, I just put this idea to rest. So my Erlang DBX idea is now lost in oblivion. Why would one need such a chip? Well Google has TPUs (Tensor Processing Units), so I thought having some dedicated Erlang chip is a cool thing to do.

As for the book, The Erlang Runtime System was cancelled, however an online copy is available.

Friday, 22 December 2017

Me at Functional Conf 2017

Some picture time. Few snaps of me with Robert Virding (Co-inventor of Erlang), Fransesco Cecarini (CTO of Erlang Solutions) and along with my friend Vipin Nair at Functional Conf 2017 held last month.

Learning Never Ends - University of York

It is official. I am now a student in the Department of Mathematics at the University of York, UK. Just thinking about it itself gives me this warm and cozy feeling. Plus a bit of anxiety because this time I have to be honest with myself and put some effort to get the degree. I applied for Masters in Mathematical Finance (Online) program. I got a conditional offer where I take the pre-sessional course, Mathematics for Quantitative Finance and depending on the results, I get to join the main course. Because my marks for my graduate degree was not exceptional and I am not in touch with Maths much for the past seven years as in I don't do intense Maths on a daily basis at work. The advantage of this particular programme is that it is self paced and I can learn and complete assignments depending on my schedule mostly, then discuss with my tutor and the exam is open ended which means I really have to understand the concepts and apply it. Also this helps me a lot because I can still continue my career and do my studies. Also doing only one thing makes me bored. This will keep me focused and will makes me busy. Plus in the end, I will be doing some rigorous learning and discussions with real Mathematicians and it comes with a degree as opposed to sitting in some corner and reading some books alone.

I did much research on available courses from many Universities and I found this to suite my taste very well because it includes Mathematics, Finance and Computing. Truth be told, I never imagined that my journey would take me to this path.

In my previous trip to UK, I visited Oxford University and nearby areas. Hopefully, my next visit to UK will be to York. Learning never ends.

Friday, 8 December 2017

Sublime Text for Clojure

Sublime Text actually changed the text editor landscape. Before it was all TextMate which was stealing the thunder. But that is somehow a begone era. In this post we will have a sneak peak on using Sublime Text 3 for Clojure development. At this time there is no Emacs to cider kind of integration right from within Sublime Text, but it is possible to write one if required, however, connecting to nrepl from command line would just do fine. Also Clojure is not Java development where you would require to add breakpoint and step code execution as things are fairly straight forward to understand as opposed to having bazillion class files when it comes typical enterprise Java projects. However if one needs, you can resort to Eclipse with Counterclockwise plugin which works just fine.

1. Have Package Control installed.
2. In the beginning there were no parenthesis. It all started with a Big Bang, after which the amount of parenthesis started expanding. Since Clojure has the eternal brackets from the beginning of universe, we need a better bracket highlighter.
Add the following config to the Sublime user settings, so that the bracket highlighter works properly and does not conflict with the editor's own matching.
{
    "auto_match_enabled": false,
    "match_brackets": false,
    "match_brackets_angle": false,
    "match_brackets_braces": false,
    "match_brackets_content": false,
    "match_brackets_square": false,
    "match_tags": false
}
This package is pretty cool as it will show the matched brackets in the gutter.
3. To keep the sanity in own pattern matching of brackets and since most Lisp programmers are borderline hippie, to commemorate that let us use Rainbowth. Once installed add the following palette in the path ~/Library/Application Support/Sublime Text 3/Packages/User/Rainbowth.sublime-settings which is taken from Emacs's package to make the highlighting easier on eyes when used in conjunction with Solarized Dark theme.
{
  "palettes": {
    "default": ["#707183", "#7388d6", "#909183", "#709870", "#907373", "#6276ba", "#858580", "#80a880", "#887070"]
  }
}
But Rainbowth needs to update the colour scheme so we need to do some extraction of the desired scheme. Let's say we are using Solarized Dark theme, then do the following.
$ cd "/Applications/Sublime Text.app"
$ mkdir -p ~/temp/subl
# Solarized theme is inside the legacy package 
$ cp "Contents/MacOS/Packages/Color Scheme - Legacy.sublime-package" ~/temp/subl
# For some other themes which are not in the above, use
# $ cp "Contents/MacOS/Packages/Color Scheme - Default.sublime-package" ~/temp/subl
$ cd ~/temp/subl && unzip "Color Scheme - Legacy.sublime-package"
$ cp "Solarized (Dark).tmTheme" "~/Library/Application Support/Sublime Text 3/Packages/"
4. Add Clojure.sublime-settings in the same directory with the below contents so that by default, the source uses two spaces, which is the standard as per style guide and since we use `-` to separate words instead of camel case or `_`, the word separator helps us to easily select symbols with double click. Else, doing a double click on a word will select up to the separator.
{
    "tab_size": 2,
    "word_separators": "./\\()\"':,.;<>~!@#$%^&*|+=[]{}`~?"
}
That's all there is specific to Clojure development.

Theming
Next we can look into few enhancements and theming to make it easy on eyes. I generally prefer dark themes as it looks cool and tends towards Solarized Dark. While generally most articles say to use a lighter background with a darker text, it depends on various other conditions like the room lighting, and of course personal preference. Refer the Dark UI section of the post for citations.

1. Install Ayu colour scheme. We will be using Ayu Mirage with Solarized Dark theme. Add the following config under user settings.
{
    "color_scheme": "Packages/Solarized (Dark).tmTheme",
    "theme": "ayu-mirage.sublime-theme",
    "ui_big_tabs": false,
    "ui_fix_tab_labels": true,
    "ui_font_size_small": true,
    "ui_font_source_code_pro": true,
    "ui_separator": true,
    "ui_wide_scrollbars": true
}
2. Additional UI settings follows. Since I got used to Monaco font from early days of OS X, I tend to stick with it instead of newer ones like Menlo.
{
    "font_face": "Monaco",
    "font_options": [
        "subpixel_antialias"
    ],
    "font_size": 16,
    "gpu_window_buffer": true,
    "highlight_modified_tabs": true,
    "scroll_past_end": true,
    "show_full_path": true,
    "translate_tabs_to_spaces": true,
    "word_wrap": false
}

Other Enhancements
1. GitGutter to see source control changes.
2. SideBarEnhancements plugin which adds more options to the side bar right click menu.
3. EDN for Clojure EDN syntax highlighting.

Once done, the editor will look elegant as in the screenshot.



Dark UI
[1] Negative contrast computer displays (dark letters on a light background) generally provide a more legible image than positive contrast displays (light letters on a dark background). Either black characters on a white background or white characters on a black background have been found to be more visible than green, yellow-orange, blue or red characters.20

[2] The greatest difference among groups is on the black & white pair: while the majority of people without dyslexia prefer it (32.67%), only 13.64% of the participants with dyslexia chose black text on white background.
Participants without dyslexia tend to prefer color pairs with a higher color and brightness contrast while people with dyslexia read faster when color pairs have lower contrasts. For instance, the color pair which was the fastest to read by the participants with dyslexia was black & creme (mean of 0.214 for the fixation duration) while black text over yellow background presents the largest fixation duration mean (0.239 seconds). Although the pair off-black & off-white is the one recommended in Web design for dyslexics (Bradford, 2011), none of the users with dyslexia selected it. This pair presents the highest fixation duration mean for the participants without dyslexia (0.193 seconds).

[3] Maureen did mention two situations when a dark background makes sense.

However, if you are using a display in a dark environment, it’s better to use a dark background as it lets you keep your eyes dark adapted. That’s why controls for airplanes and GPS units for cars switch to a dark background at night. Usually, however, the results don’t look like symbols and text floating in the darkness of space…there’s still a sense of there being a dark surface to ground the view. So the concerns above are somewhat mitigated.

I’ve seen recommendations that white on black is better for aging eyes, and for people with low vision because it reduces the amount of light scattering and distorting the image.

[4] Chief among the reasons many of you have expressed for preferring dark backgrounds is the reduced strain placed on the eyes when staring at the screen for many hours. Many developers state that light text on a dark background is easier to read over longer periods of time than dark text on a light background.

Thursday, 7 December 2017

Integration Tests with pytest

pytest is a unit testing library and coupled with requests package can be used for writing integration tests. With integration tests, couple of things we would need is a way to distinguish between environments like local, staging, production, different datacenters and run tests in all or some specific tests per env and such.

1. We need a way to pass env during test runs. We will use pytest-variables for that which makes it easy to get variables passed in from command line available to the tests. Another way is to write conftest. So our requirements.txt file will now have pytest-variables[hjson]==1.5.1 apart from other packages.
2. We will name our pytest file starting which starts with test as something like test_api.py
3. Now on writing the tests. I would recommend that the python code be modular and well organised instead of spaghetti coding. Read Modular Programming with Python by Erik Westra if you are interested. We could separate parts into different file or have things organised under class and keep it under the same file. The point is separation of concerns, readability and maintainability (especially by other programmers). A thing with OOP is that go for composition as opposed to inheritance when possible.
4. We will go with how to write pytests in an object oriented fashion rather than just writing test_ functions. Let's call the below module as test_api.py
import pytest
import logging
import requests

__version__ = "0.0.42"  # corresponds to app version
logging.basicConfig(level=logging.DEBUG)
__timeout = 90  # seconds

def get_headers(props):
    """Return header map."""
    # ...

def http_post(params, data, headers=None):
    """Send an HTTP POST request."""
    return requests.post(params['server'] + params['endpoint'], data=data, headers=headers)

class FooAPI:
    """Sample Foo API helper class.""""

    def __init__:
        pass

@pytest.mark.timeout(__timeout)
class TestFooAPI:
    """Test Foo API service."""

    pytestmark = [pytest.mark.local, pytest.mark.ny1, pytest.mark.ams1, pytest.mark.ny1_green, pytest.mark.stg1]

    @pytest.fixture()
    def foo(self):
        """Return Foo object."""
        return FooAPI()

    def gen_name():
        """Standalone function that not pytest functions."""
        pass

    def test_add_user(self, variables, foo):
        """Test adding of user."""
        log = logging.getLogger('test_add_user')
        resp = foo.create_request(variables)
        headers = get_headers(variables)
        http_resp = http_post(variables, data, headers)
        assert http_resp.status_code == 200

    @pytest.mark.ams1_beta
    def test_beta_endpoint(self, variables, foo):
        """Test beta endpoint."""
        log = logging.getLogger("test_beta_endpoint")
        data = {'users': base64.b64encode('something'), 'bar': 'baz'}
        start = time.time()
        headers = get_headers(variables)
        log.info("\nheaders: %s", headers)
        http_resp = http_post(variables, data, headers)
        rtt = time.time() - start
        log.debug("\nrtt: %s\n", rtt)
        body = json.loads(http_resp.content)
        assert body['status'] is True
        assert body['code'] == 0
        assert http_resp.status_code == 200, 'Expecting 200 OK status code.'

class TestBarAPI:
    # ....

Any class that starts with Test and any function that starts with test will be executed by pytest by default. However these behaviour can be configured. Fixtures are variables that are passed to each test as argument before it runs. There are various pytest markers. If whole pytest functions within a pytest class needs to executed by many different environments, we can pass specify them as pytestmark = [pytest.mark.en1, pytest.mark.env2, ...]. The env1, env2 correspond to env config file name. And if we need to execute specific tests against specific env then, exclude them from the above list and annotate it above that particular function as @pytest.mark.ams1_beta, which means, if the env passed via command line is ams1_beta, only then this test will execute and since the env is not specified common in the pytestmark list, other tests will be skipped. Example usecase is running sanity tests (a subset of tests) against productions at a specific interval, where we do not have to run the full testsuite. The variables argument contains the config values defined in the config json which will be available to the test methods as a python dictionary.
5. The configuration files are under config directory. Example local.json is below. The name of the file corresponds to the env which we will be passing to the pytest via command line.
{
    "server": "http://localhost:8080",
    "endpoint": "/foo/test/",
    "key": "local_key.pem",
    "cert": "local_cert.pem",
    "env": "local",
    "json": true
}
6. Tying it all together using shell script. It is better to use virtualenv. So below script does a bit of housekeeping before running the tests.
#!/bin/bash

# Integration test with pytest.

TEST_DIR=test_api
VENV_DIR=venv
CONFIG_DIR=config

function activateVirtualEnv {
    echo "Activating venv .."
    if ! hash virtualenv 2>/dev/null; then
        echo "virtualenv not found. Installing .."
        pip install virtualenv
    fi
    virtualenv $VENV_DIR
    source $VENV_DIR/bin/activate
}

function installDeps {
    echo "Installing dependencies .."
    pip install -r ../requirements.txt
}

function prepareForTest {
    echo "Running integration test .."
    activateVirtualEnv
    installDeps
}

function deactivateVirtualEnv {
    echo "Deactivating venv .."
    deactivate
}

function displayhelp {
    echo "Usage: $0 options"
    echo "where options include:"
    echo "  local    run test in local env"
    echo "  ams1     run test in Amsterdam production datacenter 1"
    echo "  stg1     run test in staging enviroment 1"
}

# $1 result xml file name
# $2 env variable name
function run {
    echo "Running test against $2"
    # -n xdist flag
    # -s capture output
    # -v verbose
    # --variables The config file
    # -m marker
    py.test -n 4 -s -v --junitxml="$1" --variables "$CONFIG_DIR/$2.json" -m "$2" test_api.py
}

# $1 results.xml
# $2 env : local, rqa2, etc. corresponds to the filename in config
# To run against all machines in staging and production, use env as stg and prod respectively
# For integration test
# ./runtests.sh results.xml local
function main {
    result="$1"
    env="$2"
    echo "result: $result"
    echo "env: $env"
    echo "config: $CONFIG_DIR"

    cd $TEST_DIR
    prepareForTest
    if [[ $env == "stg" ]]; then
        run stg1.xml stg1
        run stg2.xml stg2
        python merge_junit_results.py stg1.xml stg2.xml > $result
    elif [[ $env == "prod" ]]; then
        run ny1.xml ny1
        run ams1.xml ams1
        python merge_junit_results.py ny1.xml ams2.xml > $result
    else
        # individual
        run $result $env
    fi
    return_code=$?
    deactivateVirtualEnv
}

main $@

exit $return_code
The merge_junit_results.py can be obtained from the cgoldber gist.
To run the test we run the command ./runtests.sh ${result}.xml ${env-config-file-name}, which will check for virtualenv and such and run the test. To run the tests in parallel use xdist.
7. Sample reqirements.text file would look like
pytest==3.0.7
pytest-variables[hjson]==1.5.1
pytest-xdist==1.15.0
requests==2.13.0
pytest-timeout==1.2.0
8. To integrate the results with jenkins, it needs to know the final result of the testsuite which will be present in the results.xml file. To parse that use the code below.
#!/bin/python

"""Extract test result failure, error values for Jenkins job."""

import xml.etree.ElementTree as ET
import sys


def parse_result(file):
    """Parse the given results.xml file."""
    root = ET.parse(file).getroot()
    if root.attrib["failures"] == "0" and root.attrib["errors"] == "0":
        print "ALLTESTSUCCESS"
    else:
        print "ERROR"
        sys.exit(1)


if __name__ == '__main__':
    print "junit filename: ", sys.argv[1]
    parse_result(sys.argv[1])

Now we have a full blown integration testsuite which can be integrated with CI/CD pipeline.

For clarity, sample folder structure will look like below.
test
├── requirements.txt
├── runtests.sh
└── test_api
    ├── __init__.py
    ├── local.xml
    ├── config
    │   ├── ams1.json
    │   ├── ams1_beta.json
    │   ├── local.json
    │   ├── local_beta.json
    │   ├── ny1.json
    │   ├── ny1_beta.json
    │   ├── ny1_green.json
    │   └── vm.json
    ├── merge_junit_results.py
    ├── parse_xml.py
    ├── result.xml
    ├── test_api.py
    └── venv

Wednesday, 6 December 2017

Passcode Lock iPhone even with TouchID Enabled

I found a way to force lock iPhone that has TouchID enabled. If done as below, then the phone requires passcode to unlock even when TouchID is configured to unlock the device.
1. Enable AssistiveTouch from Accessibility.
2. And under AssistiveTouch->Custom Actions, say for Long Press set it to SOS.

Now when we need to passphrase lock, long press the assistive touch button, which will dial the emergency SOS number. Cancel the call immediately and the phone gets locked. The only problem is that it makes the emergency sound when dialling.

Or second option is to press the unlock button and tap the wrong finger five times and the phone will lock itself. Whichever is easy works. Now how would the second option work with FaceID?