Exploring Web-based Application Synchronization via Uni-directional Data Flow Architectures

Problem

There are many frameworks and solutions for implementing real-time data synchronization within client applications. However, most are extremely cumbersome and require a great degree of extra work in order to interface with existing systems. As such, one possible solution for many of these problems is adopting a uni-directional data flow as the synchronization client’s underlying architecture as it provides a natural solution for many of the problems that come with implementing synchronicity among many application clients.

Inspiration

This proposal is largely inspired by Facebook’s Flux Framework. By departing from the traditional MV* methodology, a number of complexities caused by mutable application data are removed, making the prospect of synchronization much simpler. In essence, this proposal draws on Flux’s “Uni-directional data flow model” which implements the following Observer Pattern:

Screen Shot 2015-03-29 at 12.36.57 PM

As outlined in the diagram above, actions are sent to stores containing updates to the application data. Stores maintain the state of application data and update their own state based on received actions. Components are passed an entirely updated state of the data they are interested in and render this data in some manner to the user’s screen. When any interaction involving a change in data within a Component occurs, either by a user or by application logic, a new Action containing the appropriate context for the change in data, is broadcast from the Component and the cycle repeats.

Why This Is Ideal for Synchronization

There are many characteristics inherent in a uni-directional data flow that are ideal for building a synchronization framework on top of. First off is a lack of two-way binding within the framework, or having application models listening directly to views for changes, and inversely views updating directly when a model changes without any controller interaction. As of writing this paper, this is the most common model implemented in many of today’s leading web frameworks including AngularJS created by Google and EmberJS. While this is an effective solution it often results in a lack of scoping within components of the larger system and results in complex views that require a great deal of data to maintain and render.

In a uni-directional model, component updates are architecturally expensive to implement and make passing a global data set to anywhere within the application difficult to accomplish. While this may sound detrimental, there is a major advantage in that it naturally forces developers towards implementing a modular and minimalist funnel of data down into a hierarchy of nested components. In turn this lends itself nicely to synchronization because an update to one component is contained within a small set of contained data unique to that one component.

Screen Shot 2015-03-29 at 1.33.38 PM

As seen above, the other major advantage of building synchronization on top of a uni-directional data flow is it’s action oriented manner. This naturally lends itself to a message based synchronization tools such as web sockets or RTC because the actions themselves can easily be serialized and sent to other peers. Upon receiving these foreign actions, the can easily be plugged into the the existing architecture without any additional interpretation or context making it extremely simple for the application to handle.

Initializing and Maintaining Synchronization

One of the biggest challenges with synchronization is providing initial state to a new client that wishes to synchronize with other users in an in progress application state. When dealing with a server-client architecture, the best way to accomplish this by providing the new connection with an initial payload containing the entire base action state. This is typically implemented with a single action that fires as soon as the client is ready to start processing data as designated when the root component of an application view is ready to accept data: 

Screen Shot 2015-03-29 at 2.39.57 PM

After reaching a synchronized state, the next challenge becomes remaining in such a state. Perhaps the easiest way to manage sending and receiving Actions between peered applications is by sending them to a central backing server who is then responsible for broadcasting received Actions out to all other connected applications. In the event that many events are sent to the server or received by a peer at once, both the broadcasting and execution order will be based on FIFO queue. Below is a straight forward example of what this might look like. Again, the advantage here is that the Action based system that is required in designing a simple uni-direction data flow application can easily be used to transmit changes in context and data to many peers.

Screen Shot 2015-03-29 at 8.01.00 PM

While this is the most basic of examples, using this approach is simple enough for the demands of many modern applications. One other alternative is to perform the initial render of the application on the server side which allows you to preset the initial store state directly so no initial AJAX call to load data is required.

Peer-to-Peer Connections

Things become considerably more complicated when we remove the central backing server for coordination purposes. Assuming we are using a technology such as WebRTC which is essentially a browser-to-browser communication system over sockets this is still feasible. The first major issue becomes synchronization after making an initial connection. Assuming that the new peer has been provided an ip/port combination that connects to a number of other synchronized peers the following is one solution assuming all Actions sent between peers are also appended with a timestamp at the time they were generated:

  1. Establish a connection and immediately broadcast a “PingAction” asking all peers to broadcast a global unique identifier(GUID) that they generated for themselves upon initializing their local client.

  2. Begin a FIFO queue for all actions received in the interim.

  3. Choose the first responding “PongAction” and immediately send  our a “SynchronizationAction” with their GUID specified.

  4. For the peer with the matching GUID. Immediately send back a “SynchronizationAction” with a snapshot of all application stores, and a timestamp indicating when this occurred. All other peers that receive this action that do not have the specified GUID, should simply disregard this request.

  5. After the connecting peer broadcasts the received “SynchronizationAction” to all local stores, drop all queued Actions that occurred prior to the synchronization action and then begin broadcasting those as well.

At this point the app should be in a synchronized state and should be able to receive and broadcast actions freely. There are however, two major assumptions that need to be made in a peer-to-peer system such as this. First, that a store will completely finish processing an Action before receiving another in order to avoid dirty write states, and that the timestamps appended to each action have enough specificity to avoid collisions when they are being ordered and processed. In most applications this probably isn’t going to be a major issues, but for something like gaming or with an extremely high frequency of Actions, the importance of this grows larger.

Conclusion

There are many inherent benefits in using uni-directional data flow architectures when it comes to utilizing them in synchronous environments. Whether it is in a client-server model, or in a direct peer-to-peer setting this action based architecture is ideal for circumventing many of the tedious problems that come with layering synchronization on top of traditional MVC software architectures.

Related Resources

“Flux | Application Architecture for Building User Interfaces.” Flux | Application Architecture for Building User Interfaces. Web. 30 Mar. 2015. <http://facebook.github.io/flux/&gt;.

“A Simple Library for Unidirectional Dataflow Architecture Inspired by ReactJS Flux.” Reflux. Web. 30 Mar. 2015. <https://github.com/spoike/refluxjs&gt;.

“A Free, Open Project That Provides Browsers and Mobile Applications with Real-Time Communications (RTC) Capabilities via Simple APIs.” WebRTC. Web. 30 Mar. 2015. <http://www.webrtc.org/>.

Advertisements

Creating Symfony2 Unit Tests outside of PHPUnit

The latest framework I’ve begun to experiment with is Symfony2 for PHP. At Mover we’ve integrated a few of Symfony’s modules for various bits and pieces of our internal tooling and since I’ve been rather impressed with what we’ve used so far I figured I’d give the entire framework itself a shot. As such, I’ve set out to create my first web project utilizing what seems to be a pretty well documented, explained, and exampled system.

However, I came across my first real hiccup rather shortly as a result of wanting to make my Symfony project testable outside of the defacto testing standard that is PHPUnit. As of right now I am powering my tests with Matura, a really neat test framework that a colleague of mine wrote in which I am also evaluating and hoping to test for him in the process of this experiment. Anyway, if you are using Symfony’s pre-provided abstract class “WebTestCase” you can simply extend it and Voila, you’ve got access to Symfony’s internal workings and you’re writing PHPUnit tests.

Long story short, I dove into that code, ripped out the basic pieces I’ve needed so far and if you scroll down, you’ll see the end result. A little bootstrapper class which is essentially just the Symfony bits with the PHPUnit logic ripped out.

Hopefully you find that useful, if not an alright starting point for creating your own non-PHPUnit Symfony unit tests.

Increase SEO by Linking Your G+ Account To Your Web Pages

One of the first tasks I was responsible for at Mover is linking our blog post articles to our Google Plus profiles. By doing this, Google can then index our search results using verified authors giving our posts greater credibility. Furthermore, the micro-profile of the author will show up under any indexed search results that they are the other of. This feature was released in the middle of 2012.

This is a very simple process:

1) Link to your G+ profile on your web page, and include a ref=”author” tag:

<a href=”[URL TO G+ PROFILE]” ref=”author”>Your Name</a>

Adding ref=”author” informs Google’s web crawlers that they should include the author in search results.

2) Edit your Google Plus Account

To complete the authentication loop, Google will examine your profile to see if you are indeed a contributor, preventing you or others from falsely linking authors to posts or articles. You can update your “contributions” by clicking this link.

Finally, verify that Google’s article profiler can index your newly added author information properly be entering the URL of your site into this tool: http://www.google.com/webmasters/tools/richsnippets

Devon Grand Prix 2012

This weekend was my first attempt at Road Racing. At the start of this year I told myself I wouldn’t race because it was too dangerous and I didn’t think I would be strong enough to be competitive. After I was rooked into going out to a Spring Series race I quickly changed my mind. Since the spring race I’ve come quite far. I’ve averaged about 250km/week of riding and have spent a bunch of time doing intervals and hill climbs which have had a large impact on my speed, power, and endurance.

This weekend was the Devon Grand Prix which consisted of two independent races. The first was the United Cycle Downtown Criterium Provincials in the afternoon on Saturday. The second was the Juventus Gennesee Hills Road Race on Sunday morning.

The crit race was a nerve racking experience. Unlike a road race where there is a considerable distance to travel, the crit consisted of 15 laps of about 900m. This meant that the race would last for about 20 minutes making very little room for error in terms of positioning and pacing. The cornering was intense at high speeds, but I felt comfortable and strong making tough turns in the big group and never once felt really scared about crashing. My starting spot was another story, from the get go I allowed a bunch of other riders to line up in front of me, making it very difficult to move to the front of the group. After the one parade lap, the pace car pealed away and we were off. I felt strong in the corners but was at the very back of the 35 man group. My legs felt good for the first couple laps and  I really worked hard to move myself to the front so that I was in a better position for the sprints that happened every 5 laps. I eventually made it into the top 10 as of about the 6th lap. By this point the lead pack had begun to breakaway from the main group. The guys, lead by Keegan, were really hammering and I was trying my hardest to hang on for dear life. At this point a gap of about 50m split the two packs apart. I was beginning to fade from the relentless speed and not getting on people’s wheels properly and was dropped from the lead group. I tried to fight my way back on but wasted a lot of energy in no man’s land between the two packs where I was also fighting unnecessary wind resistance. This went on for about a lap before the chase pack caught me. By this point I had went well over my red-line in terms of power output and heart rate and was really crashing hard. The weather was beautiful at 25 degrees, arguably the hottest day of riding this year. This mixed with the new black ERTC team kit I was wearing, and my insanely high heart rate left me feeling really terrible with well over 7 laps to go. I slowly was dropped from even the chase pack as I had nothing left in my legs. That, mixed with a lack of air, getting the shivers from over exertion, plus riding solo made for a rough last couple laps where I worked hard to catch and pass the other riders who had blown up early.

I came around the corner of the final lap with a Juventus guy who had been latched onto my rear wheel post getting dropped. He pulled out to sprint past me, I tried following, but the Juventus rider who crashed earlier was walking his bike across the line and neglected to look to see the last riders coming around the corner. I had to pull out at the last second to avoid putting the oblivious rider into his second crash of the day although he would have deserved it. I found out that Keegan scored enough points for 3rd overall and avoided a mechanical which was really exciting.

The take-a-ways from that race are to establish a good starting position as quick as possible, and then sit in for as long as possible while minimizing the amount of actual work you do until it becomes absolutely necessary. I also realized that once I got dropped off the front pack I should have allowed myself to join the chase pack that was working hard to bridge the gap and use them to my advantage. I was really shocked by my Garmin data and realized just how hard I had been pushing early in the race.

On Sunday I had completely different results during the road race. The race was to consist of about 72km of total riding with two massive climbs in the very middle of the ride. While my ability to produce massive power for a short time such as in the crit is not quite where it needs to be yet, I have developed quite a solid base of cycling endurance from all of the 130+ km rides I’ve done. The hill repeat nights I’ve done this year with ERTC has also made me into a decent climber, at least for Cat 5. In total ERTC fielded 5 riders for the Cat 5 race. Kyle, our strongest cat 5 rider is also a strong climber. Our game plan for the race was to stay out of trouble on the way out to the Gennesee Power Coulee, move to the front  of the pack on the first hill, and then really attack hard on the way back on the second climb shortly after. We guessed we’d need about 6-10 people to make the break away work. Assuming that I didn’t repeat the previous days performance, that was about all we hoped for and didn’t really have a plan after that.

The race started pretty calm until after the first corner at which point the guys working hard at the front began to open up the throttle. We had a strong tailwind on the way out and were doing over 40km/h a lot of the time. As it was Cat 5, there were a lot of strange accelerations where a gap would open up by about 10m in the middle of the peleton but then the guys driving the pace would back off and the groups would rejoin.  I realized after this had occurred for the 3rd time not to waste energy working hard to get onto the front group as it was just a useless waste of energy. I gradually moved into the top 10 or so riders nestling in behind Kyle so that I knew I had a non-sketchy wheel to follow. We were just far enough back so that we wouldn’t have to do any pulling and close enough that we could break out to chase down any attacks that may have occurred. Fortunately no serious attacks were mounted before the hills as I guess everyone else had the same plan as attacking on the hills.

We hit the first down hill and were flying, doing well over 60km/h. Things flattened out and then the 1km+ climb started. The strong climbers were all near the front and as opposed to waiting until the second hill, started to fly up the first hill. Not wanting to get dropped I stomped on it, cruising past other riders who weren’t climbing nearly hard enough to keep up. A group of about 15 people out of 40 made it up the hill quickly enough to separate from the lead pack. As soon as we crested the hill, our Cat 5 turn around came.  I had just enough time to get my heart rate settled down enough not to blow up and we were descending again. One thing I’ve learned is that I descend quickly. I don’t know what it is, maybe my heavy aluminium bike or being slightly heavier than most riders, but I have an ability to descend fast. On the downhill, I pulled out around the pack and settled into a an aero position and began pulling away from the front of the group. I didn’t really have plans to go off the front, but thought maybe I could make the other fast guys do some extra work to catch up to me on the decline, wasting energy for the climb. This also allowed us to separate ourselves from the chase pack that was developing as a result behind us. The hill climb was nasty, I’ve never climbed that hard before for that long. I was red-lining again and barely made it over the hill but I figured everyone else was hurting just as bad and managed to hold on.

We managed to form an echelon among the lead group and held the chase pack off for the whole race. By the end there were a couple of guys skipping pulls who looked like they were hurting bad so we tried to up the pace so that we could ruin them for the sprint. Kyle and I talked about what to do with about 7km left in the race. If their hadn’t been such a strong wind I was considering trying to ride off the front as I felt like I had better long term power than in the sprint. I decided not to risk blowing up before the finish line and ruining my chance of finishing high. The 500m mark came quickly and the group took off really far from the line. A group of about 4 people took  off and I managed to get around the guys who were maxed out to catch them. I made the mistake of not getting onto their wheels and spent about 300m in the unnecessary head wind. Kyle managed to pull away, easily pulling off to take first place in the sprint, along with another Juventus rider. I passed the other two guys who were on their tails, but with about 30m left another Juventus rider who had been drafting me pulled out and grabbed third place by about a wheel.

I was stoked on life, getting 4th place was much better than I expected, and Kyle taking 1st was awesome. We executed the game plan perfectly and everything went about as good as I could have possibly dreamed for my first race. The only thing I would have done differently would be to have moved a bit closer to the front of the break-away for the start of the sprint and then getting on a wheel until about 150m before I pull out to make my move.

Here’s the Garmin data for the race: http://app.strava.com/rides/11641433. I was really happy with being able to keep my heart rate for that long. I feel like my HR thresholds have been moving up because I have been feeling really good at higher heart rates that used to ruin me a lot quicker. I also managed to get my HR up to a new personal best of 193 on the final sprint which is the highest I’ve ever seen it on the bike.

The Pursuit of Bread Making(aka Happiness)

I’ve lately been intrigued by both my co-workers’ and training buddies’ talk of bread making and have decided to give it a shot. I love bread, its even above Cheese, Beer, Coffee, and Balsamic Vinegar on my list of foodly obsessions. I normally burn through about 2-3 loaves a week of French bread alone with Balsamic Vinegar and Olive Oil and figure that I could probably redirect the weekly cost into a steady stream of baking ingredients and great general satisfaction.

I was first intrigued with bread making over a year ago when I been making my own pizza doughs as they are much cheaper and tastier than pre bought crusts, and provide the satisfaction of making them from scratch. While this is fine, I can only handle making pizzas every so often. I have been using Peter Reinhart’s Neo-Neopolitan Pizza Dough recipe which  has served me really well so far although I am sure that there are many things I am doing incorrectly. I realized today that the version I had been using all this time was not the authentic version and called for no sugar, which I am assuming would have a large impact on how the dough turns out. Once I make it through these last 4 crusts I’ve made I’ll have to give it another shot with  the added sugar. Anyways, if you are interested in making dough from scratch pizza doughs have been fairly easy to make and quite rewarding. My only suggestions would be to invest in a pizza stone for the oven, and also corn meal to help make sliding your pizzas on and off the stone easier. On the topic of Peter Reinhart, I just finished watching his TED talk on bread making which I really enjoyed. It examined both a bit of the food chemistry behind the baking process and also the importance of bread from a philosophical/cultural point of view. I’ve linked the video below.

Anyways, to support my recipe book collecting/Amazon purchasing addictions I’ve ordered Artisan Bread in Five Minutes a Day: The Discovery that Revolutionizes Home Baking and a dough scraper to make life easier. The book was highly recommended and has an introduction that speaks about the basic techniques you’ll need to get started with bread making as well as featuring great recipes, or at least that’s what the internet is telling me. Once my gear arrives and as I learn more tips and tricks and begin cranking out loaves and I’ll probably throw pictures/what I’ve learned up on here.

As a side note I was also really intrigued by the end of the TED which taught me that you can add Spent Grains to your baking which is a by-product from home brewed beer, UH-OH! Maybe once I’ve graduated and have some more money to throw around I’ll consider it, but in the mean time, I’ll have to stick to breads.

Coronation Triathlon

Well, today I officially became a triathlete. I also found out that you can do all of the brick’s, mini-tri’s and training you want, but nothing compares to the overall hurt that you experience in a real triathlon.

The swim was about what I expected it to be. I actually did a pretty good job of predicting my overall time and got seeded with a group of people who were all pretty close in terms of speed. Our lane had no real issues with passing or turning. One really annoying feature of Peter Hemming way pool that I discovered really quickly is that it lacks the little lip around the edge of the pool. While this really has no affect on you if you can flip turn, I cannot and it made grabbing onto the wall really tricky to do my side turns. There were probably about 4 or 5 really terrible turns that I had probably cost me a couple of seconds and some wasted effort.

The swim itself wasn’t bad, I felt I paced myself pretty well and made sure that I didn’t blow up. I did get the light headed feeling that I get when I really push my distance at high pace. It’s almost like a strong head ache, but I’ve had this before so I didn’t panic. I got out of the pool after swimming my 1000m and felt horrendously bad. This being the second time I’d swam 1000m in my life after doing 100m repeats and 400m time trials for most of my most recent training it was definitely a bit of a shock. I think it was more that I hadn’t really experience the feeling of swimming hard, or that long, then pulling myself out of the pool and sprinting for T1.

My transition was surprisingly fast. I got out of the pool about half a length behind the other 2 faster guys in my lane who said they had some major race experience and beat them onto the bike course. While I didn’t practise my transitions before hand, I really spent some time and thought about what I was going to do and laid my gear out pretty well. The toughest thing to simulate is the feeling of having shaky arms from the swim and feeling like you are going to pass out/vomit. Also, getting my tri-top on was another funny scene to watch. It’s tough to do when you are soaking wet.

The bike course was fairly tough. On the slight down hill heading towards the river, there was about a 20km/h headwind which really kills some of your momentum. There was one spot where the shelter was really good and the course got a bit steeper so I really pushed hard to get up near 47km/h ish. It was surprising how many people were not taking race lines, people were pretty much everywhere but on the fastest path through the course. This worked out really well for me as I didn’t have to deal with getting around people who were in the race line. The uphill was tough to gauge. It was in that awkward spot between a hill climb and a gradual slope so you really had to work hard to find the right effort/gear to climb fast but not waste energy. Also, it felt pretty sweet pacing people with race wheels and aero bikes and aero helmets on my entry level road biking Cannondale.

Other than the feeling of T1, the run was the hardest part and definitely hurt a lot as to be expected. I was familiar with the rubbery feeling that accompanies changing from the bike to run was from all of the brick sessions I’d done. But for some reason I had gotten really tight on the front-outside of my leg between my knee and my ankle. I have felt the feeling before when running on a sloped highway so maybe the strange elevations and angles of the roads and paths caused it. It really felt like the limiting factor in my run. I was going about as fast as possible without it exploding in pain. That being said I was still around ~165bpm average on the heart rate and was still going fairly fast. The only flat part of the course was the last 1.5km.  I started feeling a bit stronger and the pain in my leg either went down or adrenalin kicked in and I was able to lay down some sub 4min/km for the last little bit. I sprinted it in to finish the race off, I think people thought I was dogging it and then picked it up at the end when really my last 1km was a sprint in comparison to most of the people I passed.

Official Splits(with transitions)

1:40:29 Derek DOWLING Edmonton 7/26   M2029 231 
Swim: 113th   21:56 
Bike 36th   47:28 32.9km/h (includes t1 and t2)        
Run: 22nd   31:06  3:54min/km

Takeaways

-Need to do some more distance work in the swim

-Once my tri bike shoes come, I need to practise the flying mount. Running down pavement in carbon soled shoes is both expensive and looks really dumb.

-There is definitely value to taping gels to the top tube of your bike, it’s hard to get at them from the side pockets of my vest

-Power Gel’s taste about 100x better than GU’s

-Need to get out of the shoes on the bike quicker, could have had a train wreck when I came flying up to the dismount line and wasted time

-Need to drink more on the bike, felt pretty dehydrated going into the run

-Need to run more, I’ve been neglecting it

-Need to brick off of road rides more, it’s a lot different than bricking off a spin bike

Also, based on my Garmin which is pretty accurate, the course was short by about 1.9km on the bike and about 0.7km on the run. The difference between 7th place and an age group podium finish was about the difference between my swim time and the average of our age group.