Speed

The product was extremely fast due to the use of carefully constructed C data structures and keeping all data in memory. Performance was over five orders of magnitude greater than the benchmark product.

Low System Requirements
Due to keeping all data in memory, the system is bound by RAM. However, through some normalisation, a typical instance for a medium sized company would consume about 15MB of RAM including the web server and database server. In memory compression could drastically reduce this if it became a bottleneck.

Event Sourcing and CQRS 
The whole system uses event sourcing and can be rewound to any point in time. Scope was left for decoupling commands and queries for CQRS but due to the speed of the system this wasn’t required at this stage.

Failsafe
Using zero-configuration networking and distributed objects any number of application instances on the same network will automatically discover each other and replicate events amongst themselves. An alternative method of forwarding events over HTTP is included to bridge different networks.

The main issues that I encountered while creating this system were in deployment. It’s actually pretty difficult to find a good way to deploy ObjC web applications, especially outside of a Mac OS X environment. On top of this, Mac OS X hosting is much rarer and more expensive than Linux hosting. I investigated many options including WebAppKit, Bombaxtic, FrothKit, WebObjects on GnuStep, SOPE, cgic, Cocotron and more. I found every option lacking. Bombaxtic and FrothKit have been abandoned, WebAppKit is too young, GnuStep is stepping into uncomfortable development territory, Cocotron lacks an NSDecimalNumber implementation and cgic feels too much like re-inventing the wheel.

For a long time I have been meaning to learn Python and/or Ruby. I decided this would be the perfect opportunity and have started porting the prototype application from Objective-C to Ruby as a learning project. I imagine that the implementation will be a few orders of magnitude slower but I will be interested to get quantitative results. It should still be significantly faster than the benchmark.

I’ve been impressed by the terseness of Ruby although I miss the static typing of the Objective-C world. I find the RubyGems system to be excellent and am using various gems for decimal numbers (FLT), http interfacing (Sinatra), CSV parsing, JSON encoding, WebSockets and Redis integration. I’ve also made use of Heroku’s free starter packages to deploy the application on the web and have been blown away at the possibilities that a platform-as-a-service offers.

For zero monetary outlay I have my application and web front-end hosted online. I have SSL and websockets for realtime updates and a free Redis instance. I can add free SMS messaging, email sending, memcached and HTML to PDF/XLS conversion. And to deploy the application all I need to do is push from my local git repository to heroku’s remote. The possibilities are staggering. To have these services for free necessarily entails them being subject to quite restrictive service levels or throughput, but to be able to create something so scalable and so easily is quite mind blowing. No longer do you require a VPS and expertise in setting up, securing and maintaining multiple pieces of infrastructure. For practically nothing (a few hundred dollars per month) you can launch your product with decent service levels once you have it all working. If it takes off and you’re in the fortunate situation to have so many users that heroku is costing too much, you can then migrate away to your own servers as there is no lock in and it’s an open stack.

The barrier to entry has been significantly lowered.

In-case Systems

As an eager beaver I turned up to my final year a week early to take part in SPARK06, an entrepreneurial course run at Bristol University. This is an excellent course, like a combination of Dragon’s Den and The Apprentice. We had two days to create a business plan before pitching to several investors. Our pitch was for a company called in-case systems and being the token electronic engineer I was the technical lead.

Above is the logo I created for our business. The premise was that a huge number of bags are lost or mislaid by airlines and other transport companies. We intended to change this by replacing the standard barcode labels applied to luggage at airports with a superior RFID label.

The advantage of RFID compared to a barcode is that they can be scanned without requiring direct line of sight of the label. Our intention was to have scanning gates/posts every 100m from check-in to cargo hold, thus allowing very precise tracking of baggage at all times. This would also be a deterrent against baggage theft.

Our business plan and technology was well received by the panel and some consideration was given to taking the idea further, perhaps by attempting to partner with a regional airline to solve the chicken-and-egg problem. Sadly further research showed that a couple of companies were already trialling such systems in Hong Kong so the idea was abandoned.

Quizzer

I’ve found that I learn well by repeating drills so I decided to write a program to allow me to make computerised multiple choice tests. This software allowed varying numbers of questions and answers, repeated harder questions more frequently, allowed embedded media (video, images, sounds) for questions and answers and kept running statistics. Rather than use this to help me study for finals, I decided to enter it into the new enterprise business competition run by Bristol university. Here is the (terrible) promotional website:

StockBook

I have always been something of an Apple nut. As a student I watched their stock keep rising to the lofty highs of around $70, thinking perhaps one day I’d be able to afford to buy some of this great company after graduation (damn it!). Initially, I used a program called BlueChip to monitor the share price as part of a fantasy portfolio but when considering buying it thought it would be a better (and cheaper!) learning exercise to re-create the functionality myself. The first versions were made in RealBasic before I re-wrote it in Objective-C and Cocoa. The core functionality is to track the value of various investments overtime using freely available market data.

Oxford Trading Analysis

Continuing on the stock market theme, I became interested in learning PHP development, neural networks, CGI and technical analysis. Throw all of this together and you get Oxford Trading Analysis (I’m originally from Oxford, UK). I wrote a neural network library from scratch in C and hooked it up to my web server using CGI (old school, I know). I used my slowly improving design skills and new found PHP prowess to build a front end website. I leveraged the work I had done in StockBook to retrieve market data from Yahoo Finance and fairly soon had a public, Google Advertising funded website.

This would allow a visitor to select a stock, and chart the last few weeks data. This data would be fed through my neural network to attempt to predict the next few days movements. All this worked swimmingly aside from two significant problems:

- The mathematics behind predicting the stock market in this way is utter BS
- My cheap web host cut CGI script execution after just a few seconds resulting in no useful output

I did however refer to this work in a later interview for a quantitative trading house, who employed me before I left uni so perhaps a worthwhile learning exercise!

MapPin

Facebook hit the UK during my penultimate year and was slowly gaining traction through some UK universities, Bristol included. It was far from dominating the field at that point in time with other networks in the UK being more popular such as WAYN (Where Are You Now?). Since I stayed on to do a Masters many of my friends had left at the end of the previous year and were off travelling the world. Often they wanted to keep in touch with people back home but found mass-emailing to be too intrusive while blogging was too technical. Additionally they didn’t always want to give out their real email address to people they met on their travels and wanted a good way to share their pictures.

I decided to create a social network to learn more PHP and client side web design. The network was called MapPin (a working title) and its focus was on travel, to address the issues encountered by my travelling friends. It was born from the code of an earlier social network I had begun and abandoned (The KissingProject – a story for another time!) but took many cues from Facebook in terms of features, unassuming style and user restrictions. Its “killer feature” was tagging people and places in photos and and using Google Maps to visualise journeys.

Eventually this project was put aside in favour of others and finals. I did later consider a re-write using CakePHP, but by this time TripAdvisor and CitiesIveVisited had become big players selling out for millions as Facebook apps. Kudos to Facebook for inventing a web based app platform!

InMyMac

When I started my geeky course on electronic engineering, I was practically laughed out of the lecture hall for turning up with an Apple iBook! Pretty much everyone else had Windows computers and thought Macs were a joke. This started to change when Apple announced they were transitioning from PPC processors to x86. More and more Macs started popping up around the university and people kept coming to me asking for the best free (students!) software to replicate some functionality from Windows.

I created the following website to catalog various applications I used frequently. This was another Linux/Apache/MySQL/PHP gem which did pay for itself with advertising revenue for a while. I also gave the code away to a friend who ran a sister site called InMyWin.

Incidentally, even the most initially Apple-hating of my geeky course mates is now a die-hard Mac user!

Globerot

Podcasts were all the rage back then having recently been picked up by Apple and integrated into iTunes. My housemate was interested in creating a satirical video podcast looking at current affairs and lamenting the state of the world. I put together a website and logo and hooked up a podcast publishing system. Sadly (fortunately) the podcast was never published and I’ve opted not to upload a video file in order to protect the guilty!

Blinky

Late in the year, with finals looming were my thoughts on revision? No. Perhaps on writing up my thesis? No. Not intending to get a job I was busily working on the prototype of my fortune-making project, Blinky! Finally using some of that electronics learned over the last few years I used an ATMEL micro controller, a bunch of shift registers and a nice LED matrix to produce a sound-reactive animated toy.

Blinky Hardware Prototype (Video)

Using a simple analogue low-pass filter it would detect the bass beat of music to change each frame. It operated in two modes, a “disco dance floor” mode with random colours generated by reading the thermal noise on an analogue pin, and an “animation mode”. Both modes are shown in the video of the working prototype linked above. I wrote an accompanying cross-platform piece of software called “Blinky Animation Studio” which allowed end-users to create and exchange new animations. The final design was intended to have square LED diffusers to give a more dance floor-like effect, as shown in the prototype website.

It’s strange recalling that this was what I intended to do after university. One of the challenges is to bring the manufacturing cost down so that profits aren’t eaten up by certifications, shipping, distribution and retail. The difficulty I faced is the many different ways this product could be put together – use a basic FPGA to reduce component count with increased component cost, discrete logic vs expensive controllers and so on. In the end I was hired by a company with overly restrictive employment contracts and Blinky was abandoned.

Stryder

Having been hired while still at university by a quantitative trading company, I decided I had better learn something about what these companies do. I decided to create a new application as an enhanced version of my StockBook app. This program, “Stryder” (not sure where all these names came from!), would have an embedded scripting language and allow the user to define an algorithmic trading strategy which could be back-tested over historical market data. The results in terms of statistics and profit were to be presented graphically. This screenshot shows an early version.

The application was written in a mix of C (electronic engineer’s favourite) and Objective-C (Apple fanboy’s favourite). Rather than make things easy on myself, I designed a custom scripting language grammar and used Flex and Bison to create an interpreter – this is what is shown in the “Rule” text box. I based my scripting language on that in a similar program by Drew McCormack called TradeStrategist. The rule shown tells the system to BUY (at the open price) if the closing price is higher than the previous day’s closing price.

As an aside, this model obviously isn’t possible in real life and the value of close(0) should be retrieved as an expected value from some indicators rather than historical data. The program downloads the historical data from Yahoo (yay for code reuse) and then runs through the date range specified working out trading signals for each day based on the rule/model entered.

Significant further work was done on this project to allow multiple rules, multiple equities, custom indicators (including neural networks), portfolio optimisation (using genetic algorithms), graphing the results, syntax-highlighting rule entry, local database caching of data and basic slippage modelling. Additional indicators, models, optimising engines and so on could be swapped in using a plugin system rather than having to re-compile the whole program. This was done as the intention was to release it to the market place. However, once again this was stifled by employment contracts. Perhaps it will be resurrected for the Mac App store in time.

iPortfolio

The iPhone was announced during my final year. Initially Apple decreed that there would be no third party native applications on the device. However the possibilities were too tempting for many developers and a bunch of people reverse engineered their own unofficial software development kit. I started writing iPortfolio against this unofficial SDK and later against the early official Betas provided by Apple after they caved to immense demand. This is essentially a mobile version of StockBook, allowing you to track your portfolio of equities on your iPhone. I doubt it compiles against the latest version of the SDK and much of the code is probably redundant given all the new APIs since then.

The morals of the story? Finish what you start and don’t sign stupid contracts!

Genetic algorithms are algorithms which mimic concepts from evolutionary biology in order to perform some search or optimisation function. First of all, let’s look at what is meant by search and optimisation by way of an example.

Imagine that you are the pilot of an airliner. The efficiency of your aircraft is related to the altitude at which you cruise. If you fly too low, the air is dense and the resulting increase in drag will cause you to use more fuel to maintain your crusing speed. Conversely if you fly too high where the air is much thinner, the engines will not have enough air to operate effectively and you will also use more fuel than is necessary to get to your destination. Somewhere between these two extremes lies an optimal altitude at which flying any higher or lower would result in more fuel being used.

Knowing this altitude is advantageous as it gives the airline an economic advantage compared to less well informed competitors. If it is using less fuel, its operating costs are lower making the business more competitive. In fact many airlines monitor the flight paths of all their airliners, and will reprimand pilots who deviate from a fuel efficient route without good reason.

Now we know why we are trying to find this altitude, we need a method to go about it. If you were to draw this relationship between efficiency (on the vertical axis) and cruising altitude (on the horizontal axis) you would end up with a graph that resembled the following:

Looking at the graph above, we can immediately see that it is shaped like a hill. By tracing a horizontal line from the top of the hill to the vertical axis, we can determine the maximum efficiency that our aircraft can obtain. If instead, we dropped a vertical line from the top of the hill to the horizontal axis, we can see at what altitude this maximum efficiency occurs – the optimum cruising altitude. The graph is described by the following function (which in this example may be provided by the engine’s manufacturers):

This equation serves as a model for our system. As we have the mathematical function which describes the altitude-efficiency relationship, elementary calculus provides us with the tool to find the optimum altitude analytically. If we differentiate the original function, we are left with a new function which describes the slope of the tangent to the original function for any given point on the horizontal axis. In plain English, this means we can find the rate at which the efficiency is changing at any point on the graph.

Plotting this equation on our existing graph gives the following:

Clearly, the line created by the new equation crosses the horizontal axis directly underneath the optimal efficiency (the “hill top”) indicated by the original function. The point where this line crosses the horizontal axis tells us the optimal cruising altitude for our aeroplane. To find this point, we solve the equation for dy/dx = 0 (or in plain English, find the altitude at which the rate of change of the efficiency is zero, where it has stopped increasing but has not yet started to decrease):

Therefore the optimum altitude for efficient flying is 36 (thousand feet). Putting this value back into our original model tells us that the efficiency at the optimum altitude is also 36 (miles per gallon). Adding the above line to our graph helps to make the relationship clear:

In this example we were searching for the optimum altitude for efficient flight. Because we had a known function relating altitude and efficiency, with a single optimal value, which was also easy to differentiate, we were able to find the optimum value using analytical methods.

In the next post we will look at some more complicated examples which are not so simple to solve!

This proved somewhat tricky to debug – was the problem in the server implementation, my code or the Javascript framework? I used a RESTful web service testing tool called RESTClient for FireFox to send some POST requests to the server manually and everything worked as expected. This was going to require some lower level debugging – time to break out the command line. I installed tcpflow, an open source TCP packet inspector which for some reason I’d never installed on this laptop. Using this tool it’s trivial to monitor the data being sent via TCP to and from the computer

In this case, my WebAppKit server was running locally. The command to monitor only those TCP packets sent via the loopback interface (and only to the port I was running the server on) was: sudo tcpflow -c -i lo0 port 12345. Within a short while I’d narrowed down the problem to a lacking implementation in the POST body-parsing code in WebAppKit. The server code expects the HTTP Content-Type header to be set to application/x-www-form-urlencoded, the default setting for HTML form data. If the header is set to anything else, the request body is skipped. An issue has been filed on the GitHub repository – now to write up a patch on my fork!