Managing Relations in Ember Data with JSON API

One decision you’re faced with when starting a new project is how you’re going to structure your API responses. If you’re using recent versions of Ember and Ember Data, then there’s a good chance you’re using the json-api specification as the basis for your api, since that works out of the box with Ember Data.

Ember Data  2.4 introduced a feature referred to as Reference Unification. The reference unification added two new methods to DS.Model instances:  belongsTo and hasMany , which should not be confused with the DS.belongsTo and DS.hasMany functions that are used to declare relations on your models.

These methods, combined with Ember Data’s adoption of json-api standard, give you some powerful tools to inspect and control related fields on your Ember Data models.

In this post, we’ll look at how some of the choices you make in your API design affect the behavior of your Ember Data models.

All of the code samples use an example set of models that look like the following models below. Specifically, we’re looking at the Book model and how you can manage the lifecycle of the related models: Author, and Chapters.

JSON API and loading relations

In our example, we want to be able to fetch a book and then fetch the related author and chapter resources.

There are two ways of defining relationships in the json-api spec. You can either provide the IDs of the related resources (the “data” method) or provide a url that the client can use to fetch the related resources (the “link” method).

Relations as “data”

The first way to return relations in your API is to return the actual “data” of the relationship. That just means you return the IDs of resources that related to the resource you requested. A json body using this “data” style looks like this:

Checking and loading “data” relations

The benefit of using this method is that it’s obvious who the author is and how many chapters there are, just by looking at the json response.  What that means for Ember is that you can inspect the IDs of your relations synchronously and without triggering any Ajax requests. Compare this to doing something like book.get(‘author.id’) which will trigger an Ajax request if the author has not been loaded yet.

Reloading “data” relations

Reloading related resources with the data method is pretty straightforward too (though it will seem a bit backwards once you get through the “links” way of doing things). To see if any relations have been added, removed, or changed, you simply reload the book model to get the new IDs.

Benefits:

– You always know if a related field has a value on the server without additional Ajax requests. There is no need to query the API for data if you know a relation is empty.

– Related models are referenced directly, so Ember Data is able to use the local store more efficiently by avoiding redundant Ajax requests.

– This is the only way to side-load relations.

Drawbacks:

– hasMany relations load each individual relation with a separate Ajax request. If you have a lot of related models, you’ll have a lot of requests fired off by your browser the first time you load those models (unless you side-load). That may cause significant performance issues.

– Reloading relations can be a bit unintuitive, because you reload the parent model instead of requesting a reload for a relation.

– You can’t reload just a single relation at a time.

– On the server, generating a list of related IDs for every API request may be prohibitively expensive and can slow down your API if not optimized properly.

Relations as “links”

Returning relations as links is the alternative to returning the IDs of the related objects.

A JSON payload using links would look like this:

Checking and loading “links” relations

The major difference from the data method is that you no longer know what the related objects are until you request them from the server. You’ll see that this has some benefits over the data method, depending on your requirements.

Reloading “links” relations

Reloading related resources when using the links attribute involves simply calling a reload on the related attribute. Ember Data has no way to cache a link reference, so every reload will trigger an Ajax request to fetch up to date data.

Benefits:

– Loading a relation will always trigger exactly one request instead of one request per related model like with the data method.

– Relations are not cached, so the related values will always be consistent with the server every time you reload.

– Reloading the parent model will not clear the relationships like the data method does.

– Allows for granular reloading of a single relation at a time

Drawbacks:

– You’re never able to tell the difference between “this book’s author has not been loaded yet” and “this book doesn’t have an author at all”. Compare this to the “data” method where the .id() method tells you if a relation is empty or just not loaded.

– You don’t know what you’re going to get until you make a request to the server. This may be a benefit if you don’t immediately care to know about a relation or if you never need that data.

– Side-loading data is not possible, because because the book model doesn’t know which of the related models are associated until a request is made.

Combining “links” and “data”

Ember Data will also work just fine if you combine the two strategies and return both link and data attributes. Returning both relationship attributes combines some of the benefits of each method while mitigating some of the drawbacks.  

A payload providing both pieces of information would look something like this:

When loading resources with both types of relationship references, Ember Data will use the “data” attribute for initial loads and the “links” attribute for reloading. I won’t include examples for the combined scenario, because they would be exact copies of the respective examples from the previous sections.

Conclusions

I think that Ember Data’s behaviour when provided with both “links” and “data” is a good indicator for the situations in which each of the two styles excel.

“Data” references are (in some cases) superior when loading data, because they allow you to side-load related resources and make use of the caching provided by the Ember Data store.

“Link” references are superior when reloading data, because all you need to do is make a single Ajax request to ensure that your client-side data is consistent with the server data.

Ember Blueprints: why they’re awesome

This is part one of a two part series. Read part two here: Ember Blueprints part 2: putting Ember Blueprints to the test

The longer title for this blog would be Ember Blueprints – More Productivity, Less Duplication, Fewer Mistakes, Less Cost.

There are numerous client-side Javascript frameworks to chose from these days, each with their own advantages and disadvantages.  Finding the right one for your organisation is going to involve looking at a lot of factors, from hiring, to number of projects, to speed of development, and much more. Here at Media Suite we predominantly work with Ember, and it’s a choice we’re very happy with.  

This is the first part of a series on building a real-world Ember Blueprint to create crud screens based off Ember models and how it can be distributed as an Ember Add-On.  While this post is about very Ember specific functionality, it’s simple enough that anyone with front-end experience can probably see what all the moving the parts are doing.

To Err Is Human

To give you a sense of scope, our team of around 15 developers typically work through five to eight Ember applications a year, in addition to maintaining existing ones.  The applications tend towards data entry and display, with lots of forms, maps, and graphs.  They’re designed to be easy to use and great to look at, but at their heart, it’s basic CRUD (Create Read Update Delete) functionality.

A lot of common patterns are re-used.  For example, we frequently have View, Edit and Add pages for a given business object – e.g. a Person.  In Ember, and in many frameworks, this is a formally a “model”.

Writing and wiring together a collection of routes, templates and components in Ember is not difficult, but there are a good number of moving parts.  It is also vital that each instance of this add/edit/new functionality is consistent, using the same nomenclature, structure and organisation.  When we churn through many projects, and offer support across a much wider number, we should play to Ember’s strengths and minimise the cognitive burden of moving from one to another.

Unfortunately, when different people do the same thing, they tend to do it in subtly different ways.  It’s also very easy to introduce trivial bugs when wiring together many modules, especially if you’re cut-and-pasting from previous examples. Code review can, should, and does catch many of these instances – but that takes time to complete, and we believe you can drive development time down while increasing quality.

To Generate Programmatically, Divine

Backend frameworks have a great line in auto-generation of behaviours and code.  Our backend framework, Loopback (https://loopback.io/) auto-generates endpoints for defined models, vastly reducing the time it takes to get a project up and running.  Django (https://www.djangoproject.com/), a Python web framework, is able to automatically create HTML input forms based on your model definitions leaving you simply to style them and add more application specific behaviours.

With automatic code generation as a starting point, we know that every developer will begin with the same structure for a common pattern.  This provides consistency.

Since the code generation has also carried out the leg-work, hours of cut-and-pasting from the last time a similar feature was used is saved.  It also removes the head-scratching when you don’t wire it together correctly.  Auto-generation of code reduces development effort and increases quality.

Ember and Angular do offer Blueprints – the ability to automatically generate code for common structures.  These are great, but limited to very common use-cases.  Fortunately Ember offers the ability to author custom Blueprints, so developers are free to create auto-generation of code for anything that is repeatable enough to be worthwhile.

(Note: Angular’s CLI was initially a clone of the Ember CLI, but custom Blueprints and Add-Ons are not yet supported.  Expect to see it in time,  though it won’t be until at least version 2.)

Docs on BluePrints

Contrary to the excellent Ember documentation, the documentation for authoring Blueprints leaves a little to be desired.  It’s pretty common to spelunk into the existing Blueprints for clues.

The existing Blueprints are in a number of different places, but the most up to date versions are here: https://github.com/emberjs/ember.js/tree/master/blueprints

and here: https://github.com/emberjs/data/tree/master/blueprints

The introduction guide can be found here: https://ember-cli.com/extending/#guide

The actual documentation for the API can be found here: https://ember-cli.com/api/classes/Blueprint.html

The latter is a great resource once you already have a good idea at what you need to do, but a little lacking for your initial foray.

The first step is to use the Ember CLI to generate a new Blueprint.

This will provide us with a directory structure like this:

The index.js file is the engine behind the Blueprint.  When ember generate model-crud person is executed on the command line, it’s going to get called and be responsible for ensuring all the correct files with the correct content are written out to the filesystem.

Adding In Blueprint Templates

Files added into the files folder (configurable via the filesPath hook)  of the Blueprint will be written into the project in the matching location.  A picture says a thousand words, so…

Everything is pretty straightforward until the  __modelToken__ and similar file names, which require a bit more explanation.

fileMapTokens Hook

The file names that look like __someToken__ are placeholders that we want to replace when the Blueprint is executed.  In this example, we’d like replace the __modelToken__ with the name of the dasherised entity name, e.g. __modelToken__ becomes person, so we have a components/person folder structure.

The fileMapTokens hook in the index.js file is where these file ‘tokens’ are defined.  There’s nothing special about the double-underscore syntax, but it is a convention, and Ember is all about consistent convention.  The code that accompanies this looks like:

The return value from this function should be an object, where the key value is the file name, or section of file name, to be replaced. The value is a string, or a function returning a string, with which to replace it.

The dasherizedModuleName is, as  might be expected, the dasherized version of the string passed into the ember generate command.  For example: ember generate model-crud SystemUser will provide an options.dasherizedModuleName value of system-user.

locals Hook

The filenames have now been written with the correct name, but the content of the files is still required.  The nice thing is, this step is also not too complicated once you understand what is going on.

Each file is an EJS (Embedded Javascript – http://www.embeddedjs.com/) template, except the output doesn’t need to be HTML, but instead will most likely be *.hbs and *.js files.  Actually any file type is allowed, the parser will simply evaluate any content within <% %> as Javascript and output other text without modification.

The shortcut syntax of <%=variableName%> will inject the value of variableName directly into the template.

Here’s the New model Route Javascript file as an example:

Which in turn, when executed with ember generate model-crud person will output the following:

The variables newMixinImport, newEditMixinName, camelizedModel and modelToken have been replaced in the file with their values.

The locals hook inside the index.js file is where we can set these values.  Below is the code required to populate the template we’ve just seen.

The locals hook must return either an object, or a Promise resolving to an object, of key/value pairs where the key is the variable name as used in the template and the value is the string that will be injected.

A More Detailed Template Example

The new Route was a very simple example, but we can look at a more complicated example below.  This is the afterModel hook from the Mixin, used by Add and Edit Routes, that fetches all of the options for the belongsTo relationships, in order to correctly populate the drop-down lists.

Which outputs the rather more simple:

availableOptions Explained

It may be you would like to add additional command line arguments to your Blueprint, which you can do using the availableOptions property on the index.js file.  For instance, if we want the model name to be different to the entity name, our command line execution could look like ember generate model-crud SystemUser –model=person.

If we execute that command without letting the Ember CLI know we want to use the –model argument, we are admonished that “The option ‘–model’ is not registered with the generate command. Run ember generate --help for a list of supported options.

The following setting will quieten Ember CLI and allow us to access the model value inside the locals hook directly on the options argument of that function.

I.e. options.model === “person” // true

It’s worth noting that this availableOptions method is not in the documentation, but examples of it’s use can be seen the “route” Blueprint here: https://github.com/ember-cli/ember-cli-legacy-blueprints/blob/master/blueprints/route/index.js

Order of Execution for Index.js Hooks

The documentation does not cover in which order the lifecycle hooks of the Blueprint fire.  Following a little digging, I’ve outlined it below.  The definition for each hook can be found in the  documentation, but a few additional notes wouldn’t go amiss.

filesPath – The path to the blueprint’s files.  The default structure is covered in more detail above.  Is called repeatedly throughout execution.

normalizeEntityName – By default, will produce a dasherized version of the string passed into the ember generate command. Is a great place to provide a default.

E.g. ember generate model-crud <thisWillBeDasherized>

locals – Returns an object or promise containing the variables that can be used in the Blueprint’s templates.

fileMapTokens – Returns an object containing the variables that can be used in template filenames.

beforeInstall – The final option to trigger something before the files are generated and written.  For example, can run other operations here in order to prepare the ground for the Blueprint.

At this point the Blueprint generates the files and writes them to the filesystem

afterInstall – Fires after the files have been written, and is a good opportunity to inform the user of any additional action they may need to take.

There are also beforeUninstall and afterUninstall hooks.  However, it is unclear from documentation and experimentation under what circumstances they fire for Blueprints.

Parsing Models is Hard

The core piece of this Blueprint is to look through an Ember model, understand what it’s attributes are, and than make sensible default decisions about how to represent that as a form.

However, I have a confession; trying to parse the model file as Javascript is hard and I was unable to do it within a reasonable timeframe.  The reason being that the Blueprint executes in a CommonJS Node environment, and the Ember model is an ES6 module.  Even using an ES6 parsing library, it’s not possible to resolve all of the model’s imported modules from within the Blueprint’s execution context.

The next approach was to try and import the generated Javascript file from the dist folder, in this example dist/my-first-blueprint.js. This is an AMD structure for modules, and none of the readily available AMD libraries appear to allow easy consumption of this file either.

Presumably there is code added to dist/vendor.js that allows the Ember Resolver to find the correctly referenced module.  However, this is where my knowledge of the Broccoli build chain and Ember internals fail me.   

The final option is to parse the Javascript file as text using pattern matching.  With the combinations of mixins, different root classes and other variances, this is definitely non-trivial and is something that has been deferred to a separate Blueprint.

In the meantime, here at Media Suite we automatically generate most of our models directly from the database schema.  We will now simply output an accompanying config json file for each model that this Blueprint process can more easily consume.  In addition, this gives us the option to specify a bit more detail about the model’s attributes and how we would like it to display – such as which field to render into a drop-down for belongsTo and hasMany relationships.

Pulling It All Together

The full code for this Blueprint can be found here: https://github.com/mediasuitenz/mediasuite-ember-blueprints/tree/my-first-blueprint

It covers the rough spec for the Model json config, and how the belongsTo relationship is handled within the templates.

A Word on hasMany and belongsTo

Typically we implement a belongsTo relationship as a searchable, drop-down list where the user can pick a single value, though in some cases it could be more appropriate to use radio buttons.  hasMany would be either the same type of drop-down but with multi-select capabilities, or checkboxes.

The UI side of this is relatively straight-forward – the json config file for each model allows us to specify which fields of the related model are relevant for display.  The saving side for hasMany is touch more complicated, as you need to save the related model, not the primary model which is being modified.  This becomes more complicated again if it is a many-to-many relationship, where you may well need to save the relationship model.

This level of uncertainty means that for time-being, hasMany relationships remain on the todo list for this Blueprint.

The full file listing for this Blueprint looks like this:

And This Is Just the Beginning

All this work gives us a pretty nice starting point, but that’s all it really is.  Right now, this Blueprint is limited to the project we wrote it in, and we can’t actually test it until we use it with a live project and a real back-end serving data.

In the follow-up post, I’ll take this code, move it into an Ember Add-On that can be installed by any project, and provide a test suite to demonstrate it behaves as it should. Read part two here: Ember Blueprints part 2: putting Ember Blueprints to the test

Inside GovHack: a non-nerd’s perspective

Happy One Year Media Suite Anniversary to me!!!

To celebrate, I just popped my Nerd Fest cherry. (I’m guilty of using the term “Nerd Fest” (self-invented) to describe any event where a large group of web developers gather.)

It seems pretty fitting, that almost one year since I joined Media Suite, I spent a full weekend immersed in one of the industry’s most famous events – GovHack. 

Yep! You wouldn’t believe that the writer of Climbing Nerd Mountain would end up volunteering to spend two full days in a windowless room filled with web developers. Yet, on Friday night, I found myself doing just that. Whether or not I contributed anything useful, well…

Me + room of devs

This is the story of one non-technical person’s GovHack experience.

Friday 5pm: We take one last swig of our traditional Friday evening drinks, and leave the Media Suite office in convoy. The Christchurch City Council building will be our base of operations until 5pm Sunday. Having done some fairly extensive research on car-parking options nearby, remortgaged our homes and donated a kidney for a parking pass, we’re all signed in by 6pm.

All competitors are based in one large, windowless room off the main building foyer. The vertical wood panelling and lack of natural light demonstrates how organisers have thoughtfully considered their target market – possibly assuming developers might feel most at home in a larger replica of their mum’s basement? To be fair, it’s the middle of winter so we’re not exactly missing much in terms of weather.

Can we talk about the toilet paper dispensers in CCC bathrooms???

By 7pm, the room has filled up. Devs dressed in polar fleece and various types of outdoorsy gear are huddled around tables, drinking from plastic jugs of Speights, laptops already open. I already feel out of my depth, even though I’m sitting at a table with my own tribe of nerds. I count 7 females in the whole room.

What is GovHack???

If you’re a web development outsider, you may well be wondering why a reputable company such as Media Suite is involved with an event named GovHack. You would not be alone. When I first joined Media Suite last year, the team was fresh from this event and I’ll admit to being slightly horrified (and a little gleeful) that I might have accidentally joined a secret organisation for hackers – like on Die Hard.

GovHack is not like Die Hard 4.0.

Perhaps unfortunately, the truth of GovHack is about as far as you can get from international espionage and cyber crime.

GovHack actually goes like this:

As I have repeatedly explained to friends and family unfamiliar with the industry, “hack” doesn’t always mean “criminal”. It’s a tough one to get your head around if you’ve spent your life watching movies like Die Hard. Hack can also mean inventing something cool, by using something in a creative way – like a lifehack. 

GovHack is an event that asks developers to take sets of data provided by the Government, and use them to create useful tools or products that can benefit society. For example, you might take data about the cost of food in New Zealand, and use it to create an app for budgeting on a low income. You know? So it’s about doing good and generating awesome ideas using government data as the starting point.

So, at GovHack, you turn up, and a huge list of datasets has been provided. Government, local government, and private sponsors have donated prizes. Groups of developers then band together, and using the datasets, try to win some of those prizes by generating the best idea/product/concept etc.

Each prize has a specific goal and set of criteria that must be met by entrants. For example, this year, Selwyn District Council asked developers to build a tool that would help coordinate disaster response efforts.

Competitors pick their categories, find the right data, then put together their idea over the course of the weekend, ready for judging. Each team also has to create a video presentation to explain their idea.

Not sure how I will contribute to such a competition, having no actual development skills, but it seemed like a good idea at the time and no-one from Media Suite tried to dissuade me (possibly because I’m too bossy to be dissuaded of much). 

8am Saturday: The Villas is the only nearby cafe we can find open at this ungodly hour. Being from Media Suite, we are accustomed to a certain standard of living, and none of us is willing to start the day without a proper cup of coffee. Plus, there are barely any developers up at this time of day so the GovHack venue is pretty subdued.

One of our teams has already picked an idea, choosing to work on the disaster response challenge. My team is not so organised. We spend most of the morning talking through concepts, only to circle back to our original idea just before lunch time. It’s good to be thorough, right? Charitable organisations have seen cash donations drop, thanks to our cash-free society, and Westpac has suggested solving this problem.

My team is made up of Andrew (Creative Director), Julia (Front End Developer) and Matt (Developer). They all bring pretty obvious skills to the table, so it’s natural to wonder what the hell I’m contributing, other than scathingly dry feedback and a strong opinion. The video presentation part of the challenge is my sweet spot. Luckily for me, before I left the media, I was given some video production training which gives me the chance to participate usefully – I hope.

Plus, I now know what an API is, so there’s that. APIs come up a lot at GovHack, and I could totally define the crap out of that acronym. To be on the safe side, I decide not to talk to anyone outside our company for the weekend, so no-one will know I am an imposter.

12pm Saturday: We decide to prototype a website/app that allows people to make regular electronic donations to any registered charity. I like the idea of doing some social good with our project. After all, that’s what we’re here for. Plus there’s capacity to use cute puppies in our video.

The afternoon is spent hashing out ideas and starting to work on our design mock-ups and video content. By this time, the room is busy and the ideas are clearly flowing. Whiteboards have gone up, there’s a sea of post-it notes floating around, and the GovHack Mentors are circling the tables offering advice and assistance.

Pizza, the staple food of any developer gathering, arrives early-evening. Work is briefly interrupted.

8am Sunday: Shit is starting to get real. Two full days seemed like heaps of time to pull this thing together, but it’s become apparent that it’ll be tight. I need to split myself between both Media Suite teams to do video production, and our team is under the pump, finishing our concept and getting all the entry information ready.

The room is much quieter today. Heads down, bums up. It’s full on completion mode. There’s a countdown clock on the big screen and more frequent trips being made to the tea and coffee station. The trash can is partially filled with empty energy drink cans and chocolate bar wrappers.

3pm: It’s almost time. One of our teams is getting their video up and the other is in final editing mode. Tempers are getting a bit frayed (mostly mine) and iMovie spontaneously decides to be difficult. Maybe the reason for the windowless room is to stop competitors from hurling their laptops out open windows.

Just as everyone is trying to upload their videos, finished teams begin to watch all the ones uploaded so far. The Wifi starts to crawl and stress levels skyrocket. It’s just a waiting game now, watching the uploading bar crawling up, percentage by percentage.

Intern James: mortified at being called to the front…

5.02pm: The video finally goes through. Our intern, James, basically wins a prize for being the youngest person there – he claims to be 18, but he could pass for a mature 15. His initial red-faced embarrassment at being centre stage is quickly erased when someone hands him a portable speaker as a prize.

Both Media Suite teams are knackered. It doesn’t sound like much, but on top of a full working week, a long weekend in front of computer screens really saps the energy levels. There’s nothing left to do now, except check out the competition’s entries, and wait for the results of the judging.

Someone gives me my first nerdy t-shirt. It has GovHack written on it #allthenerdpower. I plan to wear it in public.

Media Suite’s two teams – Frontline Rescue, and GoodCents.

When I get home, my husband asks: “So, how was it?”

I’ll let you be the judge. Take a look at our video entries below!