Exploring Web-based Application Synchronization via Uni-directional Data Flow Architectures

Problem

There are many frameworks and solutions for implementing real-time data synchronization within client applications. However, most are extremely cumbersome and require a great degree of extra work in order to interface with existing systems. As such, one possible solution for many of these problems is adopting a uni-directional data flow as the synchronization client’s underlying architecture as it provides a natural solution for many of the problems that come with implementing synchronicity among many application clients.

Inspiration

This proposal is largely inspired by Facebook’s Flux Framework. By departing from the traditional MV* methodology, a number of complexities caused by mutable application data are removed, making the prospect of synchronization much simpler. In essence, this proposal draws on Flux’s “Uni-directional data flow model” which implements the following Observer Pattern:

Screen Shot 2015-03-29 at 12.36.57 PM

As outlined in the diagram above, actions are sent to stores containing updates to the application data. Stores maintain the state of application data and update their own state based on received actions. Components are passed an entirely updated state of the data they are interested in and render this data in some manner to the user’s screen. When any interaction involving a change in data within a Component occurs, either by a user or by application logic, a new Action containing the appropriate context for the change in data, is broadcast from the Component and the cycle repeats.

Why This Is Ideal for Synchronization

There are many characteristics inherent in a uni-directional data flow that are ideal for building a synchronization framework on top of. First off is a lack of two-way binding within the framework, or having application models listening directly to views for changes, and inversely views updating directly when a model changes without any controller interaction. As of writing this paper, this is the most common model implemented in many of today’s leading web frameworks including AngularJS created by Google and EmberJS. While this is an effective solution it often results in a lack of scoping within components of the larger system and results in complex views that require a great deal of data to maintain and render.

In a uni-directional model, component updates are architecturally expensive to implement and make passing a global data set to anywhere within the application difficult to accomplish. While this may sound detrimental, there is a major advantage in that it naturally forces developers towards implementing a modular and minimalist funnel of data down into a hierarchy of nested components. In turn this lends itself nicely to synchronization because an update to one component is contained within a small set of contained data unique to that one component.

Screen Shot 2015-03-29 at 1.33.38 PM

As seen above, the other major advantage of building synchronization on top of a uni-directional data flow is it’s action oriented manner. This naturally lends itself to a message based synchronization tools such as web sockets or RTC because the actions themselves can easily be serialized and sent to other peers. Upon receiving these foreign actions, the can easily be plugged into the the existing architecture without any additional interpretation or context making it extremely simple for the application to handle.

Initializing and Maintaining Synchronization

One of the biggest challenges with synchronization is providing initial state to a new client that wishes to synchronize with other users in an in progress application state. When dealing with a server-client architecture, the best way to accomplish this by providing the new connection with an initial payload containing the entire base action state. This is typically implemented with a single action that fires as soon as the client is ready to start processing data as designated when the root component of an application view is ready to accept data: 

Screen Shot 2015-03-29 at 2.39.57 PM

After reaching a synchronized state, the next challenge becomes remaining in such a state. Perhaps the easiest way to manage sending and receiving Actions between peered applications is by sending them to a central backing server who is then responsible for broadcasting received Actions out to all other connected applications. In the event that many events are sent to the server or received by a peer at once, both the broadcasting and execution order will be based on FIFO queue. Below is a straight forward example of what this might look like. Again, the advantage here is that the Action based system that is required in designing a simple uni-direction data flow application can easily be used to transmit changes in context and data to many peers.

Screen Shot 2015-03-29 at 8.01.00 PM

While this is the most basic of examples, using this approach is simple enough for the demands of many modern applications. One other alternative is to perform the initial render of the application on the server side which allows you to preset the initial store state directly so no initial AJAX call to load data is required.

Peer-to-Peer Connections

Things become considerably more complicated when we remove the central backing server for coordination purposes. Assuming we are using a technology such as WebRTC which is essentially a browser-to-browser communication system over sockets this is still feasible. The first major issue becomes synchronization after making an initial connection. Assuming that the new peer has been provided an ip/port combination that connects to a number of other synchronized peers the following is one solution assuming all Actions sent between peers are also appended with a timestamp at the time they were generated:

  1. Establish a connection and immediately broadcast a “PingAction” asking all peers to broadcast a global unique identifier(GUID) that they generated for themselves upon initializing their local client.

  2. Begin a FIFO queue for all actions received in the interim.

  3. Choose the first responding “PongAction” and immediately send  our a “SynchronizationAction” with their GUID specified.

  4. For the peer with the matching GUID. Immediately send back a “SynchronizationAction” with a snapshot of all application stores, and a timestamp indicating when this occurred. All other peers that receive this action that do not have the specified GUID, should simply disregard this request.

  5. After the connecting peer broadcasts the received “SynchronizationAction” to all local stores, drop all queued Actions that occurred prior to the synchronization action and then begin broadcasting those as well.

At this point the app should be in a synchronized state and should be able to receive and broadcast actions freely. There are however, two major assumptions that need to be made in a peer-to-peer system such as this. First, that a store will completely finish processing an Action before receiving another in order to avoid dirty write states, and that the timestamps appended to each action have enough specificity to avoid collisions when they are being ordered and processed. In most applications this probably isn’t going to be a major issues, but for something like gaming or with an extremely high frequency of Actions, the importance of this grows larger.

Conclusion

There are many inherent benefits in using uni-directional data flow architectures when it comes to utilizing them in synchronous environments. Whether it is in a client-server model, or in a direct peer-to-peer setting this action based architecture is ideal for circumventing many of the tedious problems that come with layering synchronization on top of traditional MVC software architectures.

Related Resources

“Flux | Application Architecture for Building User Interfaces.” Flux | Application Architecture for Building User Interfaces. Web. 30 Mar. 2015. <http://facebook.github.io/flux/&gt;.

“A Simple Library for Unidirectional Dataflow Architecture Inspired by ReactJS Flux.” Reflux. Web. 30 Mar. 2015. <https://github.com/spoike/refluxjs&gt;.

“A Free, Open Project That Provides Browsers and Mobile Applications with Real-Time Communications (RTC) Capabilities via Simple APIs.” WebRTC. Web. 30 Mar. 2015. <http://www.webrtc.org/>.

TA’s Sugar Bowl Cinnamon Buns

This is a Cinnamon Bun recipe created by Travis Anderson, as a adapted from the Sugar Bowl here in Edmonton. The Sugar Bowl is known for their great cinnamon buns, and recently “released” their secret recipe. That being said, after following the instructions exactly, I couldn’t help but notice my results were nothing like there own. Anyway, I think Travis has cracked the code and come up with his own tweaks the work quite well.

Modifications to the Recipe:
Use 1440 Grams of flour
3 cups of Milk at room temperature instead of Water
3/4 TBS of salt
3 eggs at room temperature
1 1/2 TBS of instant yeast
1/2 melted butter

Steps:
Mix up the wet ingredients add 1 tbs white sugar and the yeast
leave for like 5 mins
Combine flour and salt in another bowl
Add the wet stuff and mix until integrated.
Add a bit of water if necessary, knead until a fairly firm dough forms. You may need to knead, let it relax, knead again. Let bulk fermentation go for like a half hour or so.
Mix 3/4 cup cinnamon with 1 1/4 cup brown sugar.
Melt about 1/4 cup of butter and spread it in the bottom of the two pans you will use. Sprinkle the crap out of them with the cin/sugar mix.
Divide the dough and roll out like baguettes into long ropes. Look it up on you tube if you need advice on baguette rolling.
Take room temperature butter and smear on the ropes before rolling in more cinnamon mix. Tie into knots and place in the pan. Let rise for an hour and then retard the dough overnight in the fridge.
Take out in the morning and while the oven pre-heats, melt butter and brush the tops of the buns. Sprinkle more cinimix on top and bake.

10881580_10155050985755389_7781509576794708075_n

The Fine Line Between Scope Creep and Proactivity In Software Development

Lately I’ve spent a lot of time internally debating one of my personal software design philosophies. Namely, to what extent, or if at all, should one go above and beyond the idea of MVP(Minimum Viable Product) when developing software?

That is to say, is doing the bare minimum to achieve the desired functionality perhaps even with disregard to the systems around you the best practice? Does the old saying, “If it ain’t broke, don’t fix it?” still hold water? Or is it worth taking that extra 10% of time to slightly re-architect the systems around you with consideration to that extra piece of functionality you’ve tacked on so that it doesn’t explode from the seams the next time you need to add “just one more piece of functionality”?

I’ll be the first to admit that as a fairly inexperienced developer, I have been known to be enthusiastically heavy handed when developing new systems or features. Where a chisel would have been the best tool for the job, more than once my tendency has been to immediately equip the refactor bazooka and fire at will. While I’ve built some pretty cool things as a result and made some pretty impactful changes, the void of code being released in the middle is disheartening. Not to mention the, “Hold your breath… here we deploy.” moments when the code is actually ready to ship.

As part of cleaning up my act and pursuing better development habits I’ve been making a conscious effort to refrain from these sorts of tasks and in many cases push myself in the opposite direction. Which explains how I arrived at this quandary in the first place.

In many cases it pains me when I know with just a bit of extra work, the “just get things done” changes could be made into elegant changes which improve readability, testability, and extensibility in the future. But is that really necessary or is it the case that doing this like this is simply over-architecting. Maybe it is true that if it isn’t breaking the current system or that in which you are trying to implement, then it should be left alone until it does.

My gut tells me that this isn’t the case. If you’ve been working with a code base for an extended period of time and you understand your current feature set and the foreseeable roadmap, then isn’t the extra 10% overhead now worth saving you the headache later on when the system you’ve been hacking things onto actually does explode?

I understand the arguments against this idea, or, at least trying to silo your changes to one scope of work at any given time. But from my experience, as working in a startup, “I’ll make that change in a later pull request”, really means, “It’s not going to get done.” So, with all that being said I ask, where is the line in the sand that one must draw when making these decisions?

Anyway, I’d love to hear from you, what your experiences are on this, and what I’m missing. I wrote this in about 30 minutes, probably in procrastination in studying for my American Politics Final tomorrow so hopefully it’s not too raw.

Golang Password Encryption For Apps and Websites

As one of the first exercises I’ve conducted in Golang as part of getting used to the core language/framework I’ve implemented a simple password authentication package. It uses Bcrypt for actually hashing the password with 64 bit salting created via Dev/Rand which is just standard practice really. This was more an exercise in building something practical and getting a handle on the syntax of doing some slightly more advanced things.

One thing I’ve noticed with Go, especially using this handy dandy Go Plugin, is that it’s forced me to write really clean code(at least I think so). I’ve really been trying to emphasize this lately, but Go seems to really take it to the next level, at least so far with this really simple bit of code. Anyway, I wrote the whole thing to be tested so each function has a well defined responsibility that is easy to call and check.

and the corresponding GoConvey tests:

Boom, a tested, working password authentication package! Not bad for an hour’s work.

As always, this is my first crack based on what I know, and what I read out there on the interwebs today about Golang best practices. Let me know if you see anything blatantly wrong here and I will make it not so blatantly wrong in case any poor fool uses my code.

Creating Symfony2 Unit Tests outside of PHPUnit

The latest framework I’ve begun to experiment with is Symfony2 for PHP. At Mover we’ve integrated a few of Symfony’s modules for various bits and pieces of our internal tooling and since I’ve been rather impressed with what we’ve used so far I figured I’d give the entire framework itself a shot. As such, I’ve set out to create my first web project utilizing what seems to be a pretty well documented, explained, and exampled system.

However, I came across my first real hiccup rather shortly as a result of wanting to make my Symfony project testable outside of the defacto testing standard that is PHPUnit. As of right now I am powering my tests with Matura, a really neat test framework that a colleague of mine wrote in which I am also evaluating and hoping to test for him in the process of this experiment. Anyway, if you are using Symfony’s pre-provided abstract class “WebTestCase” you can simply extend it and Voila, you’ve got access to Symfony’s internal workings and you’re writing PHPUnit tests.

Long story short, I dove into that code, ripped out the basic pieces I’ve needed so far and if you scroll down, you’ll see the end result. A little bootstrapper class which is essentially just the Symfony bits with the PHPUnit logic ripped out.

Hopefully you find that useful, if not an alright starting point for creating your own non-PHPUnit Symfony unit tests.

The DD Burger Experience

These are my go to Burgers lately. It’s a combo of tried and tested burger recipes and a rub I’ve been using a lot lately on my pork tenderloins.

Main Ingredients:

500g Lean Ground Beef, 500g Ground Pork, 3 whipped eggs, 1/4 brown sugar, 1 medium diced onion, 1 cup of diced mushrooms, 1/2 cup of bread crumbs, 1/8 cup of Olive Oil

Spices(1 Tbsp each or to taste):

Cayenne Pepper, Sweet Paprika, Salt, Pepper, Garlic Powder, Cumin, Onion Powder, Chili Powder

Money Makers(2 Tbsp each):

Liquid Smoke, Worcestershire Sauce

Combine ingredients, form your patties, makes approximately 10-12.

After forming, place in the freezer for 5-10 minutes to help stick together. Barbecue, typically on medium heat, takes about 12-15 minutes.

Harnessing Third Party Services in Tech Startups

During my tenure as an Industrial Internship Program student I learned many valuable skills and practices as an employee with Mover, a cloud storage related technology start-up based in Edmonton. Perhaps the most perceptible lesson I took away from my employment in the startup industry pertains specifically to leveraging other cloud based software services and solutions to help meet the rapidly increasing and scalable needs of a company that needs to grow as quickly as possible. CircleCi, Amazon EC2, Heroku, UserVoice, and PagerDuty are just a few examples of services that are essential to the operation, analysis, and development of the technology for migrating data between cloud services that Mover provides.

As a startup in any sector, it is important to be as effective as possible with the distribution and expenditure of your internal resources. During my time with Mover, one of the major processes through which we accomplished this was to integrate paid services to support our core technology wherever possible. There are a number of different reasons why we elected to take this route instead of building our own in-house solutions.

To begin with, it is often much cheaper to leverage even seemingly expensive outside technology than spending development resources on it. As a rule of thumb, there is a large correlation between the level of difficulty in creating a service and the price tag attached to licensing or purchasing it. The cost associated with the time it would take to design, implement, deploy, and support these features or tools can often be an order of magnitude greater than simply purchasing the tool or service in the first place. More importantly, you are not only purchasing an easy to implement solution, but also the expertise of its creators. Similar to how Mover has become experts in moving data in the cloud as a result of the time we have spent working with and optimizing our own technology, it makes sense that the same argument applies to other services that you would harness and benefit greatly from.

Another often overlooked benefit is the actual customer support that comes with purchasing or licensing these tools. When an unexpected problem occurs, it is much easier to offload your issues onto the company who created the service, who are again experts at what they do, then to deal with problem yourself. More often than not, these companies are extremely helpful and eager to solve your issues as they are themselves smaller companies who are trying to build their own customer base and reputation.

Cost and convenience aside, the most important reason for integrating and integrating outside tools wherever possible, is that allows you to spend less time working on problems that have already been solved, and more time building your core product. As a startup, you need to move as fast as possible. Any time that is not spent working on your core service has a compounding cost down the line and can become the difference between success and failure in the industry.

It’s nearly impossible to say how much time Mover continually saves with the services we have integrated. But what is extremely clear is just how vital the inclusion of many of these services has been in our technology stack, and will continue to be in enabling Mover to grow rapidly. As a result, I believe this is one of the most important philosophies I have learned and will continue to lean upon in my future as a software developer.

Tangy-Spicey Chicken Dry-Rub

I whipped up a sweet dry rub last night after I got home from a great night of riding hills in Edmonton. I wanted something super quick, super easy, that I could leave in the oven and it’d come out great.

I find the best way to mix up a meat rub(mix of spices you literally rub onto the meat) is to use a mortar and pestle. First off, you feel like a bad ass chemist mixing things together in it, and it also does a great job of breaking up any chunkies that may have formed in your spices.

I found these proportions were about perfect for about 5 chicken breasts, or about 1kg of chicken.

The concoction:

4 Tbsp brown sugar

2 Tbsp of: Sweet Paprika, Cayenne Pepper, Garlic Powder, Onion Powder, Dried Oregano, Thmye

1 Tbsp Pepper

Rub that all nice like over the chicken until evenly coated. Pre-heat your oven to 400F. Place your chicken breasts in a small lasagna pan or pie plate and pour about 4 tbsp of lemon juice over the chicken so about the botton 0.5cm of the dish is filled with lemon juice, it shouldn’t take a lot of your pieces are packed in pretty tight. Throw the chicken in, let it go for 40 minutes, flip the pieces at about the 20 minute mark, and enjoy.

 

People Watching For Self-Improvement

As I walk to and from work, university, and to get groceries, I like to “People Watch”. This isn’t nearly as creepy as it sounds, but basically as you are walking, commuting, or whatever, pay attention to the actions, postures, and general vibe of the people around you. This is something most of us don’t do, whether we are too busy texting, changing songs on our Ipod, or trying to avoid eye contact with random strangers. You’ll be amazed by how poorly 90% of people carry themselves, and with a little awareness you can avoid these same problems. I find that a lot of these have begun to help me with my own self-confidence, and as I continue to work on being aware of these things and correcting my own mistakes, I not only look better, but begin to feel better.

1) Observe the line of sight of others while they walk.

The majority of the time, especially on University campus, I notice people tend to look down as they walk. We do this subconsciously after years of avoiding eye contact with people, which I suspect stems back to our high school days. Once you realize you are constantly looking down, you will be super shocked and immediately begin to reverse this habit. First, you are now free to observe the world around you, other people, how nice the campus looks on a fall day, or a chance to make eye contact with that good looking guy or gal walking passed you. All of which are things you miss when you stare at the concrete sidewalk in front of you. Next, you’ll find that it naturally boosts your self-confidence. Walking with your head up and your shoulders back takes some time to perfect, but you’d be surprised at just how much of a difference this makes in the way you walk, and the way you feel when doing it. Add a smile in there, whether you are happy or not, and you’ll naturally start to feel like a million bucks. You’ll also begin to notice a lot more people looking at you wondering why you are in such a good mood, and probably brighten their own day in the process. A positive feedback loop if you will.

2) Look at what other’s are projecting.

Everyone projects something, and I’m not talking about stupidity or body odor.  This is a good exercise as the vast majority of the people you encounter subconsciously project a  negative self image of themselves. Next time you are on the train, bus, or at the bar, take a look at the way people hold themselves. Are they slouched over? Frowning, or do they generally look displeased? Perhaps, fidgeting or holding themselves awkwardly? You’d be surprised that just about everyone does these things. The few who don’t, who project confidence, a good posture, and a smile, will be noticeably more interesting. Why are they in such a good mood? What does he/she do? Etc. If you actively notice others doing this, you are far less likely to project a negative image yourself. If anything, say to yourself, I don’t know what their problem is, but am I ever glad that my life is not nearly as dull as theirs must be and smile about that. While being aware of your self-projection may not make a huge difference when you are walking to and from the bus everyday, practicing then makes it a habit for when it is more useful. You’ll naturally begin to broadcast a positive image  in social settings when you are trying to meet people such as at a conference or in the bar, making you much more approachable. This will allow you to focus less on you’re own body language, and more on having a good time, and interacting with others.

3) Listen to how strangers interact when they first meet.

This is an often hilarious and insightful learning experience. Far to often it sounds like people are conducting job interviews on each others and you fully expect a resume to be produced. Everyone loves to talk about themselves, but no one loves playing 20 questions with mundane canned responses to generic, canned questions.  If you notice this habit, try not to fall into the awkward trap. The trick is to not stop asking questions when you meet strangers, rather, drill deeper into interesting topics you both have in common, or that they’ve brought up that you’d like to learn more about. The best thing you can do is show interest in a persons hobbies, they’ll feel important and you’ll have probably learned a couple of cool facts about the person by the time you’re encounter is done. As you meet someone, the objective shouldn’t be to learn everything about someone in one shotgun blast of suck, but rather to pick a topic, engage at a deeper level, and slowly uncover more about the person you are interacting with as you move through related topics. Generally you will uncover a lot more about someone this way, you will build rapport at a deeper level, and often walk away having learned something completely unexpected about a topic you knew nothing about.

When you do see a conversation that is going well, pay attention! Notice how more often then not they are in a deep discussion about something in common. Generally the questions become more and more specific, or are a natural progression based on a single set of related topics. Notice the congruity. Your discussions don’t have to involve rocket science, the  meaning of life, and  they can and should be super playful as well(tone this back a bit if you are in a business situation). If you can learn to do this, you’ll find yourself not only a lot more entertained, but arguably having met a new friend, or a future business contact, that you now know a lot more about.

So there you have it, 3 simple tasks that you can do everyday. They don’t require active participation, but they do require you to observe social interaction at a deeper level. They are pretty straight forward, and hopefully you can use them to break the standard mold of our introverted, and often socially awkward generation. Like always, if you have feedback feel free to leave it below, I’d love to hear some different opinions, or similar things you like to do!

Super Simple Pork Tenderloin Rub

It’s been a while since I’ve posted anything food related here, I’ve been busy. After a tough two hour bike ride tonight, I decided that I should reward my legs with some protein. Naturally the grocery store had pork tenderloin greatly discounted so I bought a bunch. There are many different ways to marinade Pork Tenderloin, but most of those involve at least a couple of hours of refrigeration so I decided to do a rub instead, which would be pretty much ready to go instantly. I was really surprised at how simple, and yet how tasty this was so I thought I’d share it, even if Scott, my one faithful follower tries it out.

Anyway, you can play with the amounts a little bit, but I used about these proportions for 700 grams or just over 1 pound of PT.

2.5 Tbsp of Brown Sugar

1 tsp of: Chili Powder, Onion Powder, Garlic Powder, and Cumin

1/2 tsp Salt & Pepper

That’s it, throw those in a bowl and mix. I have a mortar and pestle which made it really easy to grind it into a well mixed rub with no little chunkies. The next step is to throw that in a zip lock bag with your Pork, and shake profusely. Once it looks fairly evenly coated its a simple matter of throwing them on a preheated bbq at medium/high heat. Turn every 5 minutes and pull your meat off the grill at about 18 minutes if you like it a bit rarer/juicier or 20 if you want it a bit more well done and are afraid of any pink meat. Super simple, super tasty, sweet, tangy, and a bit of a kick. If you want it a bit spicier, you could always up the Chili Powder and maybe even throw in 1 tsp of Cayenne Pepper.

The finished product, super tasty!
The finished product, super tasty!