Thursday, July 31, 2014

Install MariaDB with Custom Data Directory in Ubuntu

Here are the steps to install and configure MariaDB (v10.1.0) (a fork of MySQL) with custom data directory in Ubuntu (14.04). The MySQL server will be run as the current logged in user. You can have user as mysql, but then you will have to figure out permissions and such. If you do not want to change the default data directory then it's better to go with mysql-server package from the repository.

I was not able to get MySQL to start after changing the data directory and adding overrides in apparmor. I guess it has to do with the data directory location and permission. Next I tried installing MySQL from source so that I can specify custom directories during install time, but the build fails with this stupid ../libperfschema.a(pfs_defaults.cc.o):(.eh_frame+0x8b): undefined reference to `__gxx_personality_v0' error. But MariaDB builds and works like a champ.

1. Clone MariaDB source code from github or get the latest release version.
wget https://github.com/MariaDB/server/archive/mariadb-10.1.0.tar.gz
tar xzf mariadb-10.1.0.tar.gz
cd server-mariadb-10.1.0
2. Install dependencies.
sudo apt-get install libboost-dev libjemalloc-dev libjudy-dev bison flex libevent-dev liblzo2-dev liblz4-dev libaio-dev libpam-dev valgrind binutils-dev libatomic-ops-dev
Also check to see if any libraries are not found during the configure process. If so install the dev versions.
3. We will install MariaDB to /opt/mysql/ with data directory as say /home/username/files/mysql. The data directory corresponds to /var/lib/mysql.
cmake . -DCMAKE_INSTALL_PREFIX=/opt/mysql -DMYSQL_DATADIR=/home/username/files/mysql -DDEFAULT_CHARSET=utf8 -DDEFAULT_COLLATION=utf8_bin -DMYSQL_UNIX_ADDR=/tmp/mysql.sock
make
sudo make install
4. Add /opt/mysql/bin and /opt/mysql/support-files to .bashrc and source ~/.bashrc the paths.
5. Create data directory if not exists and install data.
cd && mkdir -p ~/files/mysql && cd files
sudo chown -R username:groupname mysql #logged in user and group
sudo chmod 777 mysql #rw to all. But since it resides inside your home dir, others cannot get into it.
# Install db
sudo /opt/mysql/scripts/mysql_install_db --datadir=/home/username/files/mysql --basedir=/opt/mysql
# Secure the installation with root password, removing test data etc.
sudo /opt/mysql/bin/mysql_secure_installation
6. Start the server and login as root.
mysql.server start
mysql -uroot -p
You can further configure mysql server using my.cnf file. Example files customized for the current installation will be located at /opt/mysql/support-files. You can copy any of the cnf file to /etc/mysql/my.cnf and make necessary change there. After that restart the mysql server. You can check if the right config file is loading by using strace.
strace mysqld
# .. output will contain
# stat("/etc/my.cnf", 0x7fff4759dc20)     = -1 ENOENT (No such file or directory)
# stat("/etc/mysql/my.cnf", {st_mode=S_IFREG|0644, st_size=4913, ...}) = 0
# open("/etc/mysql/my.cnf", O_RDONLY)     = 3
# ...
If you want to auto start MySQL on system startup, you can move mysql.server to /etc/init.d/ and update runlevel script execution.
sudo cp /opt/mysql/support-files/mysql.server /etc/init.d/mysql && cd /etc/init.d/
# enable auto start
sudo update-rc.d mysql defaults
# disable auto start of mysql
sudo update-rc.d mysql disable
# remove auto start of mysql altogether
sudo update-rc.d mysql remove
Basically having data directory on an encrypted filesystem would be very useful and is recommended (by me) and in such cases, do not auto start MySQL as the data will be unavailable unless you mount it manually. So is the case with your web sever data directory.

Tuesday, July 29, 2014

Install noip2 in Ubuntu

1. Download noip2 source
cd ~/Downloads
wget http://www.no-ip.com/client/linux/noip-duc-linux.tar.gz
tar xvzf noip-duc-linux.tar.gz
cd noip-2.1.9-1/
2. We compile it with -O3 optimization given as CFLAGS variable and install it to /opt/noip. So modify Makefile to reflect those changes (see line numbers).
CFLAGS=-O3 -Wall -g
PREFIX=/opt/noip
${TGT}: Makefile ${TGT}.c
        ${CC} ${CFLAGS} -D${ARCH} -DPREFIX=\"${PREFIX}\" ${TGT}.c -o ${TGT} ${LIBS}
3. Compile and install.
make
sudo make install
4. During the installation give in all no-ip details like username, password, no-ip domain name etc, which is similar to configuring the Windows client. The configuration file will be saved to /opt/noip/etc/no-ip2.conf. Now change the ownership of the directory to the logged in user so that no-ip can read the config file.
cd /opt
ls -la #see the ownership
sudo chown -R user:group noip  #replace user and group with your username and group most likely would be your username.
#So if your username is foo, and belongs to foo (most likely is), the run 
#sudo chown -R foo:foo noip
ls -la 
cd noip && ls -la #check ownership has been updated.
5. Setup start-stop script. You can change the prefix, optimization and such by editing the Makefile.
#Install killproc
wget ftp://ftp.suse.com/pub/projects/init/killproc-2.13.tar.gz
tar xvzf killproc-2.13.tar.gz
cd killproc-2.13/
make && sudo make install
6. Create a file /etc/init.d/noip with the following content
#! /bin/sh
# . /etc/rc.d/init.d/functions # uncomment/modify for your killproc
case "$1" in
    start)
    echo "Starting noip2."
    /opt/noip/bin/noip2
    ;;
    stop)
    echo "Shutting down noip2."
    killproc -TERM /opt/noip/bin/noip2
    ;;
    *)
    echo "Usage: $0 {start|stop}"
    exit 1
esac
exit 0
7. Now you can start and stop the service using
/etc/init.d/noip start  #start
/etc/init.d/noip stop   #stop
8. Add the service to update-rc.d to auto start at system startup.
cd /etc/init.d/
sudo update-rc.d noip defaults
9. Check is the ip is updated by logging into the noip.com website. If you are using a CNAME alias, say www of your domain name to this dynamic IP, you can check it using nslookup.
nslookup www.example.com

Tuesday, July 15, 2014

Adjust Screen Brightness in OS X using Terminal

OS X screen brightness can be adjust via Terminal using an application called brightness.
The brightness values are between 0 and 1, where 1 being the maximum brightness. Copy the file to say /usr/local/bin and run it with sudo.
sudo cp brightness /usr/local/bin/
sudo brightness 0.25

Wednesday, May 7, 2014

A Simple CSV Parser in Erlang

A simple CSV parser in Erlang. Usually splitting a line by comma would suffice unless a value itself is a string containing commas. This considers such cases as well.
parse_csv(String) -> parse_csv(String, [], [], [], false).

parse_csv([], S, Acc, [], _) -> lists:reverse(lists:map(fun(X) -> lists:reverse(lists:flatten(X)) end, [Acc|S]));
parse_csv([], S, [], L, _) -> lists:reverse(lists:map(fun(X) -> lists:reverse(lists:flatten(X)) end, [L|S]));
parse_csv(String, S, Acc, L, IsSubStr) ->
    case String of
        [$"|T] when IsSubStr =:= true ->
            % end of substring (ending quote).
            parse_csv(T, S, Acc, [$"|L], false);
        [$"|T] when IsSubStr =:= false  ->
            % beginning of a substring (beginning quote).
            parse_csv(T, S, Acc, [$"], true);
        [$,|T] when IsSubStr =:= true andalso L =/= [] ->
            % comma within a substring
            parse_csv(T, S, Acc, [$,|L], true);
        [$,|T] when IsSubStr =:= false andalso L =/= [] ->
            % comma after a substring.
            parse_csv(T, [[L|Acc]|S], [], [], false);
        [$,|T] when IsSubStr =:= false andalso L =:= [] ->
            % comma after a normal string.
            parse_csv(T, [Acc|S], [], [], false);
        [H|T] when IsSubStr =:= true ->
            % within a substring
            parse_csv(T, S, Acc, [H|L], true);
        [H|T] when IsSubStr =:= false ->
            % a normal string
            parse_csv(T, S, [H|Acc], [], false) end.
Example usage
1> c(ql).
{ok,ql}
2> ql:parse_csv("1,\"Abc, cdf\",3.0").
["1","\"Abc, cdf\"","3.0"]
3> ql:parse_csv("1,xyz,3.0").         
["1","xyz","3.0"]

Wednesday, February 26, 2014

Chrome Dev Tools For Mobile

Chrome browser has support for touch screen, mobile device emulation, accelerometer, geolocation emulation etc. Previously some of this was accessible from the Settings gear icon in the dev tools. Now it can be accessed from the console menu icon or by pressing ESC key from the dev console.



Sunday, January 12, 2014

FizzBuzz with Java 8

Using problem 1 from ProjectEuler to illustrate a few Java 8 constructs.
import java.util.stream.IntStream;

/** Project Euler Problem 1 using Java 8 idioms */
public class FizzBuzz {
    public static void main(String[] args) {
        long t1 = System.currentTimeMillis();
        long res =
            IntStream.rangeClosed(1, 10000).
            filter(x -> x % 3L == 0L || x % 5L == 0L).
            sum();
        long t2 = System.currentTimeMillis();
        System.out.println(t2-t1);
        System.out.println(res);
    }
}
No particular care is given to runtime performance. This is just to illustrate new constructs added to Java 8 like streams, filters, lambdas etc. Good to see functional programming constructs and styles getting added to mainstream languages. As of now, you can run this using Java 8 Early Access Edition with Lambda support.

Monday, January 6, 2014

Things to consider before rewriting Hybrid Mobile Apps to fully Native Apps

Usability, UI/UX Aspects
Let's focus a bit about usability points that gets often mentioned every time during such discussions, before getting into the technical aspects. From an UI/UX perspective, there are different camps of thoughts. Some says your apps' branding should be unique. Others say it must look same as the underlying platform. Now that is more about the themes and styles (I think). Usability is different from User Interface (styles, themes) and User Experience (each persons' experience in using an application). Sometimes all these sounds like hocus pocus. Anyway, I would just like to mention some general ideas.

Usability is how easily an application can be used without having to learn too much about the intricacies to get it working or to do the most basic things. So if one can have an application that behaves and have icons and buttons in same places where the rest of applications on that device have, then the user does not have to consciously think about it every time for every single application he uses. Means, if in OS X close buttons are on the left, it makes no sense to write your application that targets OS X with close buttons on the right and vice-versa for Windows. In case of mobile apps, in iOS if the convention is to have a back button on the top left, then the users generally expects any application on the platform to behaves such. Now that is the kind of thing that falls under usability. It's about following conventions, UI guidelines etc. which contributes to making any applications usable. I think, on any day this must the first priority but themes and style are close. But what's the point in having the most aesthetically pleasing application that's not very usable? Now with a complex application there are certain learning curves. Also depending on the application category, user will have to get familiarized with it if he haven't used similar applications before. Means an audio editing application is different from a social media application. Again it also depends on the familiarity of an user to similar applications. The point here is to minimize the conscious effort.

User Interface is the layer between human and machine. User interface design mainly revolves around ergonomics and psychology, be it mechanical or inside a computer (GUI). Usability and UI contributes to a user experience. It helps in creating an emotional feeling towards the application and brand. There are applications with themes and style different from other applications. Eg: Final Cut Pro (in terms to UI styles) on OS X is a different from say iTunes or Finder, but all of them follows the same Apple UIG.

User experience is generally the feeling a user associates with an application. It depends on a lot of things. It depends on user's state when using the application, previous experiences, the interaction between the application which means the responsiveness (application that makes user wait can be annoying after a certain time lapse), the ability to carry out a task. Things like fluid UI, animations, transitions, and other effects helps create a pleasurable experience. Or does it? How much of these affects user experience depends on the target audience and the application category. An audio player no matter what, has to be able to play audios without getting it stuck or having jitters. Managing audio files, having a cool looking UI, having animations and transitions etc helps improve the emotional attachment with the application. And in no case does a user like lags or slow, stuck or crashing applications. Now considering all this, how much does transitions and animations contribute? Another aspect is the long-term value a product brings and how it helps the user, which contributes to product appreciation. If a user does not want to read news then no matter what, he won't be installing a news application. Entertainment, social apps are very different from apps that targets a different niche. User expectations from a game is different from one which is purely network based, or at least the app behaves differently. Another aspect is users don't really care if the app is written in Java or C or C++ or Objective-C or Lua or C# or JavaScript and HTML5. The app should work for the user in carrying out the task the app intended to provide in seamless way. One last point, what motivated the user to use an app? What's the user's motivation? If I am using a banking app, what's my motivation or intend? To check balance, to transfer funds etc. Does the application serve that purpose? Can we make it better? What does a shopping app aim for? Can it list products in a usable way? Can the user find products and order them? Will having flashy UI help buying more products? Maybe yes, maybe no. Now these kind of things are primarily the focus of UI/UX research coupled with market research.

The point I am making is one has to consider all such factors and how much of a difference something contributes to the overall experience which will be different for different category of apps and target audiences. It's about striking a balance among various constraints. It's all a trade-off. Every choice has it's advantages and disadvantages.

Technical Aspects
Now on to more technical aspects but only in brief. First thing to keep in mind is that web technologies are a very competing platform that rivals platform specific technologies. Fully native apps has its place. If you focus only on one platform, then there is nothing wrong in having a native application. Add more platforms and everything goes exponential. Hybrid apps gives a right balance. You can have the app written using web technologies (JavaScript, HTML5 standards and APIs..) and have native code for areas where JavaScript lacks the speed to pull off. PhoneGap is one such solution that eases cross-platform application development. But keep in mind that making an application work on multiple platforms are no less challenging. Means it's not a walk in the park.

JavaScript VM Implementation
Let's look at popular JavaScript VM implementations. Chrome, Android Browser (Android v2.1 and above) and Android WebView uses V8 (AFAIK). Android 4.4 uses WebView powered by Chromium which uses Blink for rendering. V8 is the JavaScript engine from Google with a focus on performance especially with dynamic machine code generation. V8 compiles JavaScript source code directly into machine code when it is first executed. There is no intermediate bytecode or interpreter. Now think about it for a moment.
As of Android 2.2 Dalvik has JIT. The new experimental ART VM introduced in Android 4.4, does ahead-of-time (AOT) compilation to machine code during application installation.
The JavaScript engine used in Safari, Mobile Safari and UIWebView is JavaScriptCore. However the difference here is that the JavaScriptCode used by the UIWebView component which PhoneGap uses does not do JIT and is bytecode interpreted. As of now only web applications that Apple creates, websites that run in Mobile Safari and full screen websites can have JIT. This is due to a security restriction in marking the part of memory as executable which iOS doesn't allow but is required by the JIT. There is a way to allow dynamic-codesigning using entitlements on iOS, but I don't think App Store will accept such applications. But the question here is how much slow does that make? It's 2x to 5x slower that Mobile Safari. From the above we can infer that if at all a PhoneGap app gets slow the symptoms should first occur on an iPhone. No? Now there's some penalty in running a browser engine. Then again the graphics rendering is done by the native stack (CoreGraphics in iOS) and is hardware accelerated by off-loading to the GPU. Now GPU acceleration has to be turned on on native and hybrid Android apps. Current versions of Android have hardware acceleration turned on by default if targeting API version >= 14. Unless you are writing apps that's very CPU and GPU intensive the difference in having a platform specific app and hybrid (HTML5) app is not so very significant.
What does a newspaper application that retrieves all information from the server all the time benefit from having a native app? Is a PhoneGap (HTML5) app not capable to meet the demands such an application will require in terms of CPU, GPU, UI/UX perspectives? I mentioned newspaper app just as an example to point out other similar apps i.e., client-server apps that are just like websites which pull data from server which doesn't have much computations on the client apart from displaying it in a meaningful way. Now this isn't to say that such applications are easy to write. Writing cross-platform apps that goes beyond hello world has it's own challenges. This is an example of application which is for the most part network bound. If you could reduce the network delay and have tricks up the sleeve to reduce the perceived delay your app is as good as the native version. This kind of applications are not going to get any faster without having some kind of offline storage mechanism. Caching is one important thing such app can take advantage of. Maybe such app should be build as offline apps write from the start, though offline capabilities can be added at anytime if the app is well written.

A mobile app should not be programmed like a desktop browser. The devices are constrained in memory and hardware capabilities when compared to current line of desktop computers. So people often says things like infinite scroll but everyone knows that there's no such thing as an infinite memory especially on a mobile device at least as of now. Infinite something is always lazy loading and unloading. So if you are programming using brute force techniques sure more raw speed can get the app further but will eventually hit the limits. Now having to write a better version takes time and having to write lists that load and unload elements properly without much jitter requires a bit of work. This kind of scenarios appear often in other parts of the apps as well. But in the end carefully chosen techniques will only make the app function well be it native or web or whatever. The advantage of a hybrid app is that one can call into native code for performance intensive computations if the communication overhead is less than the computational overhead.

Writing and maintaining native apps for multiple platforms are no easy task. Support different screen sizes, devices, etc. especially on Android devices are equally tiresome be it native or web. HTML5 introduces media query for this kind of tasks. But it requires some thought and CSS, HTML mastery. Nothing is easy.

An Overview of Current State of Web Applications
Now this has nothing to do with mobile browsers at present. But you will get an idea about how competing and serious the web platform is getting. Before writing further watch the Unreal Engine 3 in Firefox with asm.js video and come back. Now that's just the beginning. Okay, Chrome hasn't added asm.js yet. Now asm.js is more like assembly. It's more like an optimization setup. In traditional sense is what I mean, i.e., writing critical areas of C code in assembly for performance gains. You don't write assembly right from the beginning. It's done at a later point in time. Now writing efficient code, using correct algorithm etc. should be done on any day and isn't what optimization mainly refers to. You can read on emscripten, asm.js etc from the links and there's no point in repeating it here. Another cool project is the JavaScript PC Emulator which can boot Linux kernel right inside your browser. See for yourself. It took 81.141s to boot inside Iceweasel 17.0.9 running on netbook with Atom processor N450 with 1GB RAM. The point is speed of browser are increasing and applications you can write using JavaScript (or any other language in general) is limited only by ones own creativity and the native, browser performance gap is getting closer. But it's not quite there yet. But I am optimistic about JavaScript VM and browser stack implementations. Again, this has nothing to do with current mobile apps, just that JavaScript and Web Technologies are serious enough that one can create amazing applications using them. Quoting Atwood's Law: "any application that can be written in JavaScript, will eventually be written in JavaScript".

jQuery Mobile and PhoneGap
If you do a quick search about PhoneGap and jQuery Mobile, most of the posts mention that jQuery Mobile is slow. Now there is nothing in jQuery Mobile or jQuery that can slow down webapps. At work the team (always it's a team effort) which includes me started using jQuery Mobile from 1.0 release on wards. Initially there were some problems with the page transitions which occasionally caused jitters. We had added some CSS fixes but then upgrading it to newer version removed such problems. One area that I still have problems is with multi-select non-native lists in 1.2.0 version. When the number of items increases the selection is getting slow. Something to do with the private refresh function on the widget(?) I don't know, I haven't looked much into it. Then this problem was solved by having a custom dialog window with checkboxes. Also we had the select like list item with select bubbles added to mimic the select widget. It's faster than the non-native select menu. Apart from these two, to this day I haven't experienced any serious problems with jQuery Mobile. Now the application has 14,618 lines of unminified hand written JavaScript code of which around 10-15% would be comments giving around 10,000 lines of code. The point here is the app has passed its hello world state. (I'm not mentioning the app as I would like to be a bit anonymous and for other reasons). Now if box shadows, border radius, gradients etc are taxing for the GPU, then removing them would work.

Mobile App Performance Checklist
Before taking any drastic steps first try to identify what the problem is. These are in no way comprehensive, just some point from my mind.
1. First identify the nature of the application. See if the network is contributing to the delay. Is the delay in the perception? You can reduce the delay feel by displaying the page first and then updating the content or the display page with cached content and then update the DOM with newly fetched data etc. Having client side caching, adding offline behavior to parts of the app etc. would be beneficial. Any application that requests data from server for every single tap will benefit a lot from this kind of implementation. Now this is the same be it native or web. There is no magic solution that will make this work on a native application. Or is there any?
2. Next see if any algorithms can be improved, or tasks that can be done on the server.
3. Send only the necessary fields that are used in the client from the server.
4. Check server response time for each API call.
5. Assuming that you have a fast client side template rendering and if the templates are not compiled, compile it before hand.
6. Have a build script to compile client-side templates, use optimizing compilers like closure compiler which minify and optimizes JavaScript, HTML and CSS. Now to use that effectively you have to follow their usage guides and annotations.
7. Optimize images.
8. Use CSS sprites sheets for icons and resources used in the app.
9. Split code into modules and load only necessary parts on app startup. Load only when necessary.
10. Preload data, script etc. in cases where appropriate.
11. Unload DOM nodes if the pages added to the document increases after certain point. Unload it to local store or indexed db so that it doesn't have to be re-rendered again.
12. Use tap events and stop event propagations if done with the event.
13. Use event delegation on large list items instead of binding events to each individual item in the list.
14. Follow good coding conventions, style guides, best practices etc. Have a good and clean code base always helps.
15. Run jslint or jshint and try to stick to it. But you don't have to religiously adhere.
16. Move compute intensive tasks to native stack and call it from JS as it's a hybrid app if the call time is less than compute time.
17. Use setTimeout where appropriate to queue tasks. JavaScript is basically single threaded but the threading model is different. It's async with event driven. Basically there is nothing much in the JS or browser that blocks the UI. It's build from grounds up in that fashion. Networks calls especially XHRs does not block the UI. (Think about Node.js). I think native apps have slight advantage here if the app require it.
18. You can use web worker, if required. But Android support is lacking in older versions like 2.3, which is still being used.
19. Upgrade the libraries the project uses for bug fixes and enhancements. Use jQuery 2.x branch which removes support for older browsers if the app does not require them.
20. Looking into DOM trees and other micro-optimizations should be done as a last resort. Now this isn't to say that one shouldn't think about it.

Tooling is an area where mobile web apps lacks a bit when compared to platform specific instrumentation tools. There is weinre (WEb INspector REmote). For most part this would suffice. Then if you are playing around too much with object creation, and your apps depends on GC behavior, then you must think about the requirements before hand.

Now the choice is your own. My coding philosophies are irrelevant. But this should clear up some of the misconceptions surrounding mobile app development.

Rewriting to native apps makes sense if the application is in the early stages of development where the time spend isn't all that much and if we find that the goals cannot be met with the implementation.

Read More
1. Mobile Application Development: Web vs. Native by Andre Charland, Brian LeRoux. ACM Queue, 2011. (The article is a bit old, but there are some valid points in there. NB: The mention on JavaScript VM implementations are not that concrete).
2. The Mature Optimization Handbook by Carlos Bueno. Facebook 2013.