Exploring Web-based Application Synchronization via Uni-directional Data Flow Architectures

Problem

There are many frameworks and solutions for implementing real-time data synchronization within client applications. However, most are extremely cumbersome and require a great degree of extra work in order to interface with existing systems. As such, one possible solution for many of these problems is adopting a uni-directional data flow as the synchronization client’s underlying architecture as it provides a natural solution for many of the problems that come with implementing synchronicity among many application clients.

Inspiration

This proposal is largely inspired by Facebook’s Flux Framework. By departing from the traditional MV* methodology, a number of complexities caused by mutable application data are removed, making the prospect of synchronization much simpler. In essence, this proposal draws on Flux’s “Uni-directional data flow model” which implements the following Observer Pattern:

Screen Shot 2015-03-29 at 12.36.57 PM

As outlined in the diagram above, actions are sent to stores containing updates to the application data. Stores maintain the state of application data and update their own state based on received actions. Components are passed an entirely updated state of the data they are interested in and render this data in some manner to the user’s screen. When any interaction involving a change in data within a Component occurs, either by a user or by application logic, a new Action containing the appropriate context for the change in data, is broadcast from the Component and the cycle repeats.

Why This Is Ideal for Synchronization

There are many characteristics inherent in a uni-directional data flow that are ideal for building a synchronization framework on top of. First off is a lack of two-way binding within the framework, or having application models listening directly to views for changes, and inversely views updating directly when a model changes without any controller interaction. As of writing this paper, this is the most common model implemented in many of today’s leading web frameworks including AngularJS created by Google and EmberJS. While this is an effective solution it often results in a lack of scoping within components of the larger system and results in complex views that require a great deal of data to maintain and render.

In a uni-directional model, component updates are architecturally expensive to implement and make passing a global data set to anywhere within the application difficult to accomplish. While this may sound detrimental, there is a major advantage in that it naturally forces developers towards implementing a modular and minimalist funnel of data down into a hierarchy of nested components. In turn this lends itself nicely to synchronization because an update to one component is contained within a small set of contained data unique to that one component.

Screen Shot 2015-03-29 at 1.33.38 PM

As seen above, the other major advantage of building synchronization on top of a uni-directional data flow is it’s action oriented manner. This naturally lends itself to a message based synchronization tools such as web sockets or RTC because the actions themselves can easily be serialized and sent to other peers. Upon receiving these foreign actions, the can easily be plugged into the the existing architecture without any additional interpretation or context making it extremely simple for the application to handle.

Initializing and Maintaining Synchronization

One of the biggest challenges with synchronization is providing initial state to a new client that wishes to synchronize with other users in an in progress application state. When dealing with a server-client architecture, the best way to accomplish this by providing the new connection with an initial payload containing the entire base action state. This is typically implemented with a single action that fires as soon as the client is ready to start processing data as designated when the root component of an application view is ready to accept data: 

Screen Shot 2015-03-29 at 2.39.57 PM

After reaching a synchronized state, the next challenge becomes remaining in such a state. Perhaps the easiest way to manage sending and receiving Actions between peered applications is by sending them to a central backing server who is then responsible for broadcasting received Actions out to all other connected applications. In the event that many events are sent to the server or received by a peer at once, both the broadcasting and execution order will be based on FIFO queue. Below is a straight forward example of what this might look like. Again, the advantage here is that the Action based system that is required in designing a simple uni-direction data flow application can easily be used to transmit changes in context and data to many peers.

Screen Shot 2015-03-29 at 8.01.00 PM

While this is the most basic of examples, using this approach is simple enough for the demands of many modern applications. One other alternative is to perform the initial render of the application on the server side which allows you to preset the initial store state directly so no initial AJAX call to load data is required.

Peer-to-Peer Connections

Things become considerably more complicated when we remove the central backing server for coordination purposes. Assuming we are using a technology such as WebRTC which is essentially a browser-to-browser communication system over sockets this is still feasible. The first major issue becomes synchronization after making an initial connection. Assuming that the new peer has been provided an ip/port combination that connects to a number of other synchronized peers the following is one solution assuming all Actions sent between peers are also appended with a timestamp at the time they were generated:

  1. Establish a connection and immediately broadcast a “PingAction” asking all peers to broadcast a global unique identifier(GUID) that they generated for themselves upon initializing their local client.

  2. Begin a FIFO queue for all actions received in the interim.

  3. Choose the first responding “PongAction” and immediately send  our a “SynchronizationAction” with their GUID specified.

  4. For the peer with the matching GUID. Immediately send back a “SynchronizationAction” with a snapshot of all application stores, and a timestamp indicating when this occurred. All other peers that receive this action that do not have the specified GUID, should simply disregard this request.

  5. After the connecting peer broadcasts the received “SynchronizationAction” to all local stores, drop all queued Actions that occurred prior to the synchronization action and then begin broadcasting those as well.

At this point the app should be in a synchronized state and should be able to receive and broadcast actions freely. There are however, two major assumptions that need to be made in a peer-to-peer system such as this. First, that a store will completely finish processing an Action before receiving another in order to avoid dirty write states, and that the timestamps appended to each action have enough specificity to avoid collisions when they are being ordered and processed. In most applications this probably isn’t going to be a major issues, but for something like gaming or with an extremely high frequency of Actions, the importance of this grows larger.

Conclusion

There are many inherent benefits in using uni-directional data flow architectures when it comes to utilizing them in synchronous environments. Whether it is in a client-server model, or in a direct peer-to-peer setting this action based architecture is ideal for circumventing many of the tedious problems that come with layering synchronization on top of traditional MVC software architectures.

Related Resources

“Flux | Application Architecture for Building User Interfaces.” Flux | Application Architecture for Building User Interfaces. Web. 30 Mar. 2015. <http://facebook.github.io/flux/&gt;.

“A Simple Library for Unidirectional Dataflow Architecture Inspired by ReactJS Flux.” Reflux. Web. 30 Mar. 2015. <https://github.com/spoike/refluxjs&gt;.

“A Free, Open Project That Provides Browsers and Mobile Applications with Real-Time Communications (RTC) Capabilities via Simple APIs.” WebRTC. Web. 30 Mar. 2015. <http://www.webrtc.org/>.

TA’s Sugar Bowl Cinnamon Buns

This is a Cinnamon Bun recipe created by Travis Anderson, as a adapted from the Sugar Bowl here in Edmonton. The Sugar Bowl is known for their great cinnamon buns, and recently “released” their secret recipe. That being said, after following the instructions exactly, I couldn’t help but notice my results were nothing like there own. Anyway, I think Travis has cracked the code and come up with his own tweaks the work quite well.

Modifications to the Recipe:
Use 1440 Grams of flour
3 cups of Milk at room temperature instead of Water
3/4 TBS of salt
3 eggs at room temperature
1 1/2 TBS of instant yeast
1/2 melted butter

Steps:
Mix up the wet ingredients add 1 tbs white sugar and the yeast
leave for like 5 mins
Combine flour and salt in another bowl
Add the wet stuff and mix until integrated.
Add a bit of water if necessary, knead until a fairly firm dough forms. You may need to knead, let it relax, knead again. Let bulk fermentation go for like a half hour or so.
Mix 3/4 cup cinnamon with 1 1/4 cup brown sugar.
Melt about 1/4 cup of butter and spread it in the bottom of the two pans you will use. Sprinkle the crap out of them with the cin/sugar mix.
Divide the dough and roll out like baguettes into long ropes. Look it up on you tube if you need advice on baguette rolling.
Take room temperature butter and smear on the ropes before rolling in more cinnamon mix. Tie into knots and place in the pan. Let rise for an hour and then retard the dough overnight in the fridge.
Take out in the morning and while the oven pre-heats, melt butter and brush the tops of the buns. Sprinkle more cinimix on top and bake.

10881580_10155050985755389_7781509576794708075_n

The Fine Line Between Scope Creep and Proactivity In Software Development

Lately I’ve spent a lot of time internally debating one of my personal software design philosophies. Namely, to what extent, or if at all, should one go above and beyond the idea of MVP(Minimum Viable Product) when developing software?

That is to say, is doing the bare minimum to achieve the desired functionality perhaps even with disregard to the systems around you the best practice? Does the old saying, “If it ain’t broke, don’t fix it?” still hold water? Or is it worth taking that extra 10% of time to slightly re-architect the systems around you with consideration to that extra piece of functionality you’ve tacked on so that it doesn’t explode from the seams the next time you need to add “just one more piece of functionality”?

I’ll be the first to admit that as a fairly inexperienced developer, I have been known to be enthusiastically heavy handed when developing new systems or features. Where a chisel would have been the best tool for the job, more than once my tendency has been to immediately equip the refactor bazooka and fire at will. While I’ve built some pretty cool things as a result and made some pretty impactful changes, the void of code being released in the middle is disheartening. Not to mention the, “Hold your breath… here we deploy.” moments when the code is actually ready to ship.

As part of cleaning up my act and pursuing better development habits I’ve been making a conscious effort to refrain from these sorts of tasks and in many cases push myself in the opposite direction. Which explains how I arrived at this quandary in the first place.

In many cases it pains me when I know with just a bit of extra work, the “just get things done” changes could be made into elegant changes which improve readability, testability, and extensibility in the future. But is that really necessary or is it the case that doing this like this is simply over-architecting. Maybe it is true that if it isn’t breaking the current system or that in which you are trying to implement, then it should be left alone until it does.

My gut tells me that this isn’t the case. If you’ve been working with a code base for an extended period of time and you understand your current feature set and the foreseeable roadmap, then isn’t the extra 10% overhead now worth saving you the headache later on when the system you’ve been hacking things onto actually does explode?

I understand the arguments against this idea, or, at least trying to silo your changes to one scope of work at any given time. But from my experience, as working in a startup, “I’ll make that change in a later pull request”, really means, “It’s not going to get done.” So, with all that being said I ask, where is the line in the sand that one must draw when making these decisions?

Anyway, I’d love to hear from you, what your experiences are on this, and what I’m missing. I wrote this in about 30 minutes, probably in procrastination in studying for my American Politics Final tomorrow so hopefully it’s not too raw.

Golang Password Encryption For Apps and Websites

As one of the first exercises I’ve conducted in Golang as part of getting used to the core language/framework I’ve implemented a simple password authentication package. It uses Bcrypt for actually hashing the password with 64 bit salting created via Dev/Rand which is just standard practice really. This was more an exercise in building something practical and getting a handle on the syntax of doing some slightly more advanced things.

One thing I’ve noticed with Go, especially using this handy dandy Go Plugin, is that it’s forced me to write really clean code(at least I think so). I’ve really been trying to emphasize this lately, but Go seems to really take it to the next level, at least so far with this really simple bit of code. Anyway, I wrote the whole thing to be tested so each function has a well defined responsibility that is easy to call and check.

and the corresponding GoConvey tests:

Boom, a tested, working password authentication package! Not bad for an hour’s work.

As always, this is my first crack based on what I know, and what I read out there on the interwebs today about Golang best practices. Let me know if you see anything blatantly wrong here and I will make it not so blatantly wrong in case any poor fool uses my code.

Creating Symfony2 Unit Tests outside of PHPUnit

The latest framework I’ve begun to experiment with is Symfony2 for PHP. At Mover we’ve integrated a few of Symfony’s modules for various bits and pieces of our internal tooling and since I’ve been rather impressed with what we’ve used so far I figured I’d give the entire framework itself a shot. As such, I’ve set out to create my first web project utilizing what seems to be a pretty well documented, explained, and exampled system.

However, I came across my first real hiccup rather shortly as a result of wanting to make my Symfony project testable outside of the defacto testing standard that is PHPUnit. As of right now I am powering my tests with Matura, a really neat test framework that a colleague of mine wrote in which I am also evaluating and hoping to test for him in the process of this experiment. Anyway, if you are using Symfony’s pre-provided abstract class “WebTestCase” you can simply extend it and Voila, you’ve got access to Symfony’s internal workings and you’re writing PHPUnit tests.

Long story short, I dove into that code, ripped out the basic pieces I’ve needed so far and if you scroll down, you’ll see the end result. A little bootstrapper class which is essentially just the Symfony bits with the PHPUnit logic ripped out.

Hopefully you find that useful, if not an alright starting point for creating your own non-PHPUnit Symfony unit tests.

The DD Burger Experience

These are my go to Burgers lately. It’s a combo of tried and tested burger recipes and a rub I’ve been using a lot lately on my pork tenderloins.

Main Ingredients:

500g Lean Ground Beef, 500g Ground Pork, 3 whipped eggs, 1/4 brown sugar, 1 medium diced onion, 1 cup of diced mushrooms, 1/2 cup of bread crumbs, 1/8 cup of Olive Oil

Spices(1 Tbsp each or to taste):

Cayenne Pepper, Sweet Paprika, Salt, Pepper, Garlic Powder, Cumin, Onion Powder, Chili Powder

Money Makers(2 Tbsp each):

Liquid Smoke, Worcestershire Sauce

Combine ingredients, form your patties, makes approximately 10-12.

After forming, place in the freezer for 5-10 minutes to help stick together. Barbecue, typically on medium heat, takes about 12-15 minutes.

Harnessing Third Party Services in Tech Startups

During my tenure as an Industrial Internship Program student I learned many valuable skills and practices as an employee with Mover, a cloud storage related technology start-up based in Edmonton. Perhaps the most perceptible lesson I took away from my employment in the startup industry pertains specifically to leveraging other cloud based software services and solutions to help meet the rapidly increasing and scalable needs of a company that needs to grow as quickly as possible. CircleCi, Amazon EC2, Heroku, UserVoice, and PagerDuty are just a few examples of services that are essential to the operation, analysis, and development of the technology for migrating data between cloud services that Mover provides.

As a startup in any sector, it is important to be as effective as possible with the distribution and expenditure of your internal resources. During my time with Mover, one of the major processes through which we accomplished this was to integrate paid services to support our core technology wherever possible. There are a number of different reasons why we elected to take this route instead of building our own in-house solutions.

To begin with, it is often much cheaper to leverage even seemingly expensive outside technology than spending development resources on it. As a rule of thumb, there is a large correlation between the level of difficulty in creating a service and the price tag attached to licensing or purchasing it. The cost associated with the time it would take to design, implement, deploy, and support these features or tools can often be an order of magnitude greater than simply purchasing the tool or service in the first place. More importantly, you are not only purchasing an easy to implement solution, but also the expertise of its creators. Similar to how Mover has become experts in moving data in the cloud as a result of the time we have spent working with and optimizing our own technology, it makes sense that the same argument applies to other services that you would harness and benefit greatly from.

Another often overlooked benefit is the actual customer support that comes with purchasing or licensing these tools. When an unexpected problem occurs, it is much easier to offload your issues onto the company who created the service, who are again experts at what they do, then to deal with problem yourself. More often than not, these companies are extremely helpful and eager to solve your issues as they are themselves smaller companies who are trying to build their own customer base and reputation.

Cost and convenience aside, the most important reason for integrating and integrating outside tools wherever possible, is that allows you to spend less time working on problems that have already been solved, and more time building your core product. As a startup, you need to move as fast as possible. Any time that is not spent working on your core service has a compounding cost down the line and can become the difference between success and failure in the industry.

It’s nearly impossible to say how much time Mover continually saves with the services we have integrated. But what is extremely clear is just how vital the inclusion of many of these services has been in our technology stack, and will continue to be in enabling Mover to grow rapidly. As a result, I believe this is one of the most important philosophies I have learned and will continue to lean upon in my future as a software developer.