Using Native SQL in Loopback

Quick Loopback Overview

For those of you not familiar with Loopback, it is a web framework, built upon Express.js designed primarily for RESTful APIs.  It favours configuration over coding, and can very quickly have you up and running with standard REST end-points for common CRUD actions.

It’s not without it’s downsides, but it does a lot of the boilerplate coding for you and we use it extensively within Media Suite.  For more detail, check out their website.

The Case For Native SQL

Loopback supports a number of different database back-ends, including NoSQL databases such as Mongo, and more traditional relational databases like PostgreSQL.  Within Media Suite, we exclusively use PostgreSQL.

Unfortunately, Loopback’s flexibility with regard to database engine is also a problem for us.  Its query syntax, for the most part, limits itself to the lowest common denominator, and as a result we’re left bereft of Joins.

And sometimes you really want to use a Join.

I know, we can get a super set and filter it down, but relational databases are really good at this kind of operation; they’ve been doing it for a long time.  As swift as the Node runtime has become, it can’t, and shouldn’t compete in this space.

If we want to limit our query by values on other models, there are a couple of options:

1) Create a View and then map a Loopback model to this View.  It’s a great strategy for read-only features like a Search, but is not practical if you want to use the results to populate an existing model already mapped to a database table.

2) Write SQL, execute it and hydrate the Loopback models with the results by hand.

This blog post is going to examine the second of these options.  Some of the code in here is specific to the PostgreSQL connector, though the same pattern will be appropriate for any relational database.

Getting Access to the Connector

  • Go to the working GitHub repo
  • Go to this file: server/models/book.js
  • Note: This code is working as of Loopback 2.22.0.  As the examples use private method calls, future updates may invalidate some of the syntax.

To access any of  methods required for running SQL, we need to get hold of the database connector.  

If we deconstruct this a little bit, what are we doing?

  • The app is the main loopback object
  • The dataSources represents the data sources that are declared in the datasources.json (or equivalent js file)
  • db is the name of our chosen data source, in this case the PostgreSQL database
  • Finally the connector is the object that contains the useful methods for executing the SQL and managing the results.

Further reading: 

Running the Query

Now we’ve got access to the Connector, this bit is really easy. For the sake of this example, let’s assume a simple query:

To execute that SQL, we simply call the execute method on our connector:

Loopback is still heavily callback based, and since we’re quite deep into its API there is no promise equivalent of this method.

Hydrating the Models

We now have resultObjects; an array of objects, with each object representing one row of results from the query.  Each row contains all the fields specified in the query, in this example from the book and author tables.

Warning: If you have columns of the same name in each table, only one field will be present in the JavaScript object.  One table’s data will be lost.  This is a real problem you’ll have to manage and design for in your database schema.

We have to turn the data we get back from the query into the correct structure for hydrating the Loopback model.  We can do that using the fromRow method on the connector.

The fromRow function’s first argument is a string representing the Loopback model that you want the data in the correct format for, and the second a single row of raw data returned from the database.  Then you can hydrate a new model instance, as below:

Putting these bits of code together, we can end up something like this:

Further reading – see Data Mapping Properties for how the data mapping is managed.  It’s normally not required to manually add these details, but here is where you can implement non-default options.

What About Related Models?

Within Loopback’s standard query syntax, we can add an include to retrieve related models, but with this approach, that’s not an option. You’ll note in the SQL there were two models’ worth of data returned.  If you have unique field names, it is simple as passing the one result object (rawRowData in this example) into the fromRow method for each model.

It is then required to explicitly link the two models, as Loopback can’t infer this relationship simply from matching up Primary and Foreign keys in the model’s data.

And that’s it!  Your models are hydrated, linked and ready to be sent back via your API.
Check out the GitHub repo mentioned above to see the code all together.

 

JavaScript Idioms (Part 1)

In this article we’ll be look at some fundamental Javascript “idioms”.

What is an Idiom?

An idiom is a pattern, often repeated and well understood by a community of programmers. By using idioms we can communicate our intent clearly, avoid pitfalls, and gain a better understanding of the big ideas behind a language. In this article we focus code patterns.

We’ll avoid formatting, naming, and other items you can find in a style guide.

Below there are a number of code examples using ES2016 syntax. The values of variables are shown in comments that look like this:

// -> true

Comparison

Use ===  or !== .

==  can be confusing, use ===  to avoid headaches.

If you want to look at the bullet you’re dodging, see here.

Gotcha
JS compares objects and arrays by reference, not by their values
This results in some surprising behaviour:

There is one case where  ==  and  !=  are used. Instead of writing:

We can write:

This will check for both null, and undefined.

When writing your own code that might not return an object, prefer returning undefined over null.

Convert a number, string, object reference to a boolean

Because of implicit type conversions in JS we can negate (!) a value to convert to a boolean, then negate that again to see if the original value coerces to true or false.

Implicit checking

Javascript programs often take advantage of implicit type conversions to shorten tests, eg:

This can sometimes cause issues, eg:

  • If 0 is a valid number of attendees
  • If you’re checking for an empty array or object, !![]  and !!{}  both return true

To check if an array is empty, check people.length .

To check if an object is empty, check Object.keys({}).length .

In your own code you can use explicit checks (eg: a.length >= 0 ). Either way, it’s still useful to be clear on what a value may return when coerced.

Summary: false , Empty String ( ""),    , null , undefined , and NaN  all are all false in if statements, everything else, eg: -1 , "0" , "false" , [] , {}  are interpreted as true.

Check if an object reference is defined

Setting a default when we are unsure

When we are unsure if a variable storing an object is defined, we use the or operator ( || ) to specify a default, this works because !!undefined === false.

This checks the value of firstPartyGuest , if there isn’t one, then I am the lucky winner!

Further reading and next steps

While many style guides contain some idioms (eg: https://github.com/rwaldron/idiomatic.js/ and https://google.github.io/styleguide/javascriptguide.xml), I recommend using a linter plug-in / auto-formatter in your favourite editor for style issues instead and focus on the code itself (eg: https://github.com/feross/standard)

Douglas Crockford covers a similar issue, that is how we can write reliable, predictable Javascript, in his talk here:

To cement these concepts in your mind I recommend https://www.codewars.com/. It gives you small exercises and then a ranked list of other user’s answers.

Happy coding!

Top 10 Ramda Functions at Media Suite

The latest version of Ramda contains 235 individual functions.

To help you on your path to functional greatness, here are our picks for the top 10 functions from the Ramda library. We’ve used these across three large, complex web applications developed at Media Suite. The method used for collecting this data is tracked here.

A key concept in functional programming is writing functions that work on other functions. One of the most important examples of this is the concept of currying.

Consider this example:  We have a regular function add  which we pass as an argument to R.curry . The result of this is a function which can not be called without passing in all of the defined parameters. When addCurried(1)  is called, we get a version of the  add function where the first parameter is always the value “1”. All functions in the Ramda library are curried by default.

R.propEq

This function takes three parameters: a property name (string), a value and an object. It checks if the property name on the object is equal to the value provided. Most of the time, this function is used in a filter clause  or if clause .

R.merge

This function is similar in function to Object.assign. It expects two object arguments and returns an object. The behaviour is to merge all fields from the right hand object into the left hand object. This is typically used within a “map” function to apply fields to another object.

R.isNil

Part of writing good reusable functions is checking that the correct parameters are used. R.isNil  lets us check if the value is nil or undefined. In our projects over the last year we have used this function 79 times.

R.filter

This works like Array.prototype.filter, but with more currying and switched around parameters to what you may expect, coming from other utility libraries like lodash. It expects two parameters. First, a function which takes one parameter and returns truthy or falsy values. Second, an array. We use this function frequently to get the right array of things before passing that array into another function for further processing.

R.pluck

At number four out of 10, we used the pluck function, 111 times. It takes a string and array of objects as parameters. The return is an array of values picked out from each object in the input array.

R.map

Map takes the same parameters, and returns the same type of value as filter. They are best friends. Often the output of a map is the input to a filter (or vice versa). It was used 200 times.

R.pipe

In the previous nine functions I have demonstrated, we have made some neat little functions. Using currying, these functions now operate on a single value. To use a food as an analogy, R.pipe  is the bread holding our tasty functions together to make a delicious sandwich.

In the previous example we used getArea , getSquares  and finally console.log  to print the results. The problem with nesting functions like this is that our eyes have to scan in an un-natural way to figure out how the data flows through functions.  We have to find the value shapes , then scan each function to the left to understand the how the function operates on that parameter. The following example shows how we can create the same function by using pipe.

Displaying Promises in EmberJS Templates

Displaying Asynchronous Data in Templates

In Ember, a situation we commonly run into is needing to display asynchronous data in a template. A lot of the time this works really well, for example: Ember has some smarts around displaying related models within templates.

The below example shows a simple library application displaying authors and their published books:

1

As you can see, despite the fact that getting the books is an asynchronous traversal of a relationship, the template renders correctly once the books have been fetched.

This is all well and good, but when you want to display some asynchronous data that isn’t just properties on a related model it gets more difficult.

For example, what if we wanted to asterisk authors that had published in the last two years? This requires us to get the related books and run a little logic. This seems like a useful function to have on our model so we can reuse it throughout our application.

However, now we can no longer use that value directly in the template – in fact, it requires some restructuring of our application. We could possibly pass in the resolved promise as part of our route model, however this will be clunky as we have a list of authors. Another approach is to create a separate component that is responsible for displaying the author. This component could call hasPublishedRecently  and set a property on the component once the promise resolves.

 

This works, but requires us to add a new component as well as a chunk of extra code just to display the resolved promise value.

Using DS Promises

Ember does provide another way to do this. Rather than a function on the model we can use a computed property and if it returns a special type of object it can be used directly in the template. Ember Data provides these two classes:

They use the PromiseProxyMixin to give the Ember objects extra properties and methods that the templates can work with.

To demonstrate, we can adapt the previous example to use a computed property and return a PromiseObject  instance instead of a plain promise.

Now we can use it directly in our template:

2

Notice we now need to use the content  property of the object to display it in the template? If you want to ensure that the promise has resolved you can also use the properties isPending  or isSettled . See the PromiseProxyMixin for others.

For example, if we had a promise telling us if the book was available or on loan, we could wrap it in an unless isPending  block so nothing will show until the result was known:

Drawbacks of DS Promises

Forgetting to use content or isPending properties

One drawback of this approach is that you now need awareness of whether you are dealing with a promise or a value in your templates. One convention we’ve used in our projects is giving the computed properties resolving to promises a Promise  suffix.

 

This adds a few more keystrokes but does make it much clearer in templates when dealing with these properties.

Boilerplate

This approach still requires some boilerplate code. It needs the promise to be wrapped in a DS.PromiseObject.create()  call and accessing the content property in the template. It may be possible to remove this with a helper that can take a regular promise and abstract away the details of PromiseObject  and optionally wrap the template code within an unless isPending  block.

Although we haven’t tried it, this library also looks promising and solves the same problem https://github.com/fivetanley/ember-promise-helpers.

References

To see a working example of this code, you can check out the repository at https://github.com/mediasuitenz/promise-template-blog-post.

A Day in the Life of an Ember Concurrency Task…

If you don’t know ember-concurrency…Recommended reading: This article by Alex Matchneer or watch this video on the ember concurrency website. Play with these examples.

Right, for those who want to crack on, let’s get into it!

Not everything is as it seems

Nominally, the .drop()  task modifier in ember-concurrency indicates that no more than one instance of a task may ever be running once at a time. However, you might have missed this crucial sentence from a page deep down in the Route Tasks page of the documentation:

“ember-concurrency tasks are scoped to the lifetime of the object they live on, so if that object is destroyed, all of the tasks attached to it are cancelled.”

The important takeaway here is the first bit: “ember-concurrency tasks are scoped to the lifetime of the object they live on”. i.e. ember-concurrency tasks are scoped to object instances, not classes.

As you’ll see below, you can actually have multiple instances of a task running simultaneously, despite having applied a modifier (drop, enqueue, restartable, keepLatest) to prevent that from happening. This applies to all of the task modifiers, but we’ll just stick with drop in our examples to keep things simple.

Take the following component (view the full twiddle example here):

What’s going on?

In the application.hbs template, we’ve rendered out the component twice. Since ember-concurrency tasks are scoped to instances (in this case, instances of a component), each rendered component gets its own, unique instance of myTask .

Each instance of myTask  keeps its own completely independent state, and that enables us to perform myTask  on both components simultaneously. To demonstrate: if we were to double-click Button One we see the text Running pop up next to Button One (but not Button Two) and we only see a single pair of logs in the console:

  • Start One
  • Done One

That’s ember-concurrency doing its job for us by preventing the task from being run a second time while the first is still running.

However, the same rule does not apply when we click Button Two immediately after Button One. When doing that, we see Running show up next to both buttons, and the following in the console:

  • Start One
  • Start Two
  • Done One
  • Done Two

Enter the Singleton

If necessary, we can get around this duplication effect by defining our tasks in singletons. In Ember, routes, controllers and services are all singletons, meaning only a single instance of any given route, controller, or service is created. Once created, they are never destroyed.

The obvious connection to ember-concurrency tasks is: if I had defined myTask on a route instead of in a component, it would not have been duplicated. In that case, we truly would only ever have one instance of myTask  running at a time, regardless of how many times or combinations of buttons we pressed.
And there you have it! This behaviour by ember-concurrency is definitely a sane default and likely the most commonly used behaviour. On the rare occasion you do need multiple sibling components to share task state, it could cause some unexpected bugs in your application if you’re not paying attention.