We use the word “entity” because the word “object” is too loaded - it has too many connotations from Object-Oriented Programming (OOP), and it also has a separate grammatical meaning in terms of the “subject” of a verb versus the “object” of a verb.
Entities are everywhere in software. MySQL tables, backbone.js models, Protocol Buffers, JSONs, Plain Old Java Objects, XML documents, Haskell records - as programmers we spend an inordinate amount of time working with the entities that matter to our systems. It is no coincidence that these entities play a huge role in the events which occur in and around our software too.
There is a lot of confusion around the role of entities within events - even to the extent of one analytics company arguing that entity data is completely distinct from event data. In fact nothing could be further from the truth - as we’ll soon see, our events consist of almost nothing but entities. But before we look at the event-entity relationship, we need to understand how entities interact with time.
All objects exist in a moment of time
Amy Tan, The Bonesetter’s Daughter
In the real world as in software, entities change over time. If we can view our entities as consisting of properties, we can divide these properties into three approximate buckets:
The exact taxonomy is not set in stone - for example, a person’s height will start as a frequently changing property, become a static property in adulthood, and then switch to an infrequently changing property as she grows older. Nor do these properties necessarily change in particularly directional or meaningful ways over time: my geographical location has patterns in it (such as my daily commute), but there’s no overarching trend across all the data.
Let’s imagine that Jack is playing a mobile game, and saving his progress as he goes. Jack is an entity, and the mobile game is another entity. Across three distinct save game events, Jack’s internal properties change:
To put it another way: it was a slightly different Jack who saved his game each time. If we want to analyze and understand these save game events, being able to review the “version of Jack” who enacted each event could be hugely important. How can we do this?
What matters in life is not what happens to you but what you remember and how you remember it
Gabriel García Márquez
We are surrounded by software that helps us to model entities in one form or another. Separately, we’ve learnt that the entities we care about change over time. Presumably all of this entity-modeling software makes it easy to record how our entities are slowly (or rapidly) changing over time? Actually, most of it does not.
Most systems that represent entities - databases, schema languages, serialization protocols and so on - have no concept of time as a dimension. These systems most often simply track the current state of each entity: accordingly as developers we are expected to update in place the existing data when some property changes. If Jack’s email address changes, then we must update the existing email address value in his record in the players table. In many systems, it is as if Jack’s email address was always the new one.
This isn’t such a problem for permanent or static properties like Jack’s first language - but it’s a real pain for properties that change infrequently or frequently, such as his email address or location. Various software approaches have emerged to tackle this problem:
Each of these approaches to Change Data Capture has different pros and cons. And more importantly, many companies won’t have any Change Data Capture in place. So how do we make sure that we can cross-reference our events against the relevant entities as they existed at the time of the event?
Storage is cheap and getting cheaper. Networks are getting faster all the time. If we care about the entities relevant to a given event, why don’t we just interrogate their state at the exact moment of the event and record these entities’ state as part of our event? To put it another way: let’s snapshot our entities’ state inside of our events, and thus record Jack’s exact properties each time he saved his mobile game.
There are some distinct advantages to this approach:
Leaving aside the language of “custom contexts”, what we are doing here is recording the state of two entities - our web page and our user - at the exact moment of our page view event. These are entity snapshots in all but name.
In the next section, we will reconcile this new idea of entity snapshots with our earlier event grammar.
Back in the 2013 blog post, we identified six discrete building blocks from human language which would make up the semantic structure of our events. We introduced them with this diagram:
We explained the six building blocks thus:
- Subject, or noun in the nominative case. This is the entity which is carrying out the action: “I wrote a letter”
- Verb, this describes the action being done by the Subject: “I wrote a letter”
- Direct Object, or simply Object or noun in the accusative case. This is the entity to which the action is being done: “I wrote a letter”
- Indirect Object, or noun in the dative case. A slightly more tricky concept: this is the entity indirectly affected by the action: “I sent the letter to Tom”
- Prepositional Object. An object introduced by a preposition (in, for, of etc), but not the direct or indirect object: “I put the letter in an envelope”. In a language such as German, prepositional objects will be found in the accusative, dative or genitive case depending on the preposition used
- Context. Not a grammatical term, but we will use context to describe the phrases of time, manner, place and so on which provide additional information about the action being performed: “I posted the letter on Tuesday from Boston”
Knowing what we now know about entities, we can make two simplifications to this model:
Making these changes results in events that take this form:
This updated event grammar is set out in this diagram:
As we predicted at the start of this blog post: our events now consist of almost nothing but entities. Apart from our Verb, our event consists only of entity snapshots, each tagged with the grammatical term that relates it to the rest of the event.
So far this is all a little theoretical. At Snowplow most of our applied event modeling is done in Iglu-compatible self-describing JSON, so why don’t we sketch out what our revised event grammar could look like in this format?
Imagine that we are trying to model an in-game event where a player (Jack, perhaps) sells some armour to another player. In the following self-describing JSON, we represent this event as a set of annotated entity snapshots:
Note that all of the schema URIs are illustratory - don’t go looking for the underlying schemas in Iglu Central! There are a few things about this approach which stand out:
This is a good place to pause and take stock. I’ve attempted to move the foundational Snowplow event grammar concepts forward by re-framing them in terms of entity snapshotting, a powerful approach which we are de facto already implementing via Snowplow custom contexts. I look forward to getting the Snowplow community’s thoughts and responses on the above - with the aim of evolving these ideas still further. Do please leave any and all comments in the Feedback section below!