Data.js

Sun Jan 29 2012

Data Manipulation and Graph Persistence for Node.js and the Browser.

Introduction

Data.js is a data manipulation and persistence framework for Javascript. It has been extracted from Unveil.js, and is now being developed in the context of Substance, a web-based document authoring and publishing engine. It took some inspiration from various existing libraries such as the Google Visualization API or Underscore.js. 

You can report bugs and discuss features on the GitHub issues page, on Freenode IRC in the #_substance channel, post questions to the Google Group, or send tweets to @_substance.

With Data.js you can:

  • Query, manipulate and persist data on the client (browser) or on the server (Node.js) by using exactly the same API. 
  • Make fast computations (like grouping and aggregating data, filtering) or implement your own transformers
  • Synchronize nodes with a data-store (CouchDB) and handle conflicts that may occur in a multiuser scenario.
  • Subscribe for graph updates in realtime (using stateful Websockets for transport)

Think of it as ActiveRecord for the browser, but different as it is using a graph rather than tables.
Data.js is based on generic data formats that you can use for representing data of your particular domain. 

  • Data.Hash —A sortable hash data-structure
  • Data.Graph — A data abstraction for all kinds of linked data
  • Data.Collection — A simplified interface for tabular data (uses a Data.Graph internally)

Download and Installation

  • Latest Release (0.4.1) — 49kb, Uncompressed with Comments
  • Production Version (0.4.1) — 20kb, minfied (currently not available, please minify yourself)
The Source Code is hosted on Github, and you can view an annotated version to make yourself familiar with the internals. Data.js depends on Underscore.js.

Installation Browser

Download the latest version or pick a specific release. Don't forget to include a recent version of Underscore.js.

Installation Node.js

npm install data
var Data = require('data');
var items = new Data.Hash({a: 123, b: 34, x: 53});

Data.Hash

A Data.Hash is basically an extension to regular Javascript objects or arrays. It provides hash semantics (random access) but also array semantics (Data.Hash instances are ordered). There are a numerous  convenience functions you can make use of. Data.Hash instances form the building block of Data.js and are heavily used throughout the framework. They come in handy when used stand-alone too. Let's create and work with a Data.Hash:

Usage

var populations = new Data.Hash();
populations.set('austria', 8402908);
populations.set('germany', 81742000);
populations.set('usa', 310955497);

// Access like a hash
populations.get('austria') // => 8402908

// Access like an array
populations.at(0) // => 8402908

// Perform a sort by value
populations.sort(Data.Comparators.ASC) // => [ 8402908, 81742000, 310955497 ]

// Aggregate
populations.aggregate(Data.Aggregators.AVG) // => 133700135

Methods

get    hash.get(key)

Get the value at a given key. For example: populations.get('austria') 

at    hash.at(index) 

Get a value at a given index. For example: populations.at(0) 

set    hash.set(key, value)

Set a value at a given key. For example: populations.set('france', 65447374) 

del    hash.del(key) 

Delete an entry at a given key.

first    hash.first()

Get first value. 

last    hash.last()

Get last value. 

index    hash.index(key)

Returns for a given key the corresponding index. For example: populations.index('austria') // => 0

key    hash.key(index)

Returns for a given index the corresponding key. For example: populations.key(0) // => 'austria'

each    hash.each(iterator)

Iterate over values contained in the Data.Hash. For example: 

populations.each(function(val, key, index) { 
  alert(val);
});

values    hash.values()

Returns an ordinary Javascript Array containing just the values.
For example: populations.values() // => [8402908, 81742000, 310955497] 

keys    hash.keys()

Returns an ordinary Javascript Array containing just the keys. 
For example: populations.keys() // => ['austria', 'germany', 'usa'] 

toArray    hash.toArray()

Convert to an Array containing key-value pairs. Internally used by sort().
For example: populations.toArray() // => [{key: 'austria', value: 8402908}, {key: 'germany', value: 81742000}, ...] 

toJSON    hash.toJSON()

Serialize as JSON. For example:  populations.toJSON() // => {"austria": 8402908, "germany": 81742000, ...} 

map    hash.map(iterator)

Produces a new Data.Hash by mapping each value through a transformation function (iterator). iterator's arguments are (value, key, index).

var mappedItems = items.map(function (item) {
  return item/1000000}).toJSON();
// => {austria": 8.402908, "germany": 81.7420, ...}

select    hash.select(iterator)

Looks through each value in the hash returning a new Data.Hash containing all entries that pass a truth test (iterator). For example:

var selectedItems = items.select(function(value, key, index) {
  return value >= 10000000;
}).toJSON();
// => {"germany": 81742000, "usa": 310955497}

sort    hash.sort(comparator)

Performs a sort using a comparator function. Provided comparators compare values, but certainly you can pass your own iterator.

var sortedItems = populations.sort(Data.Comparators.DESC).keys();
// => ["usa", "germany", "austria"]

intersect    hash.intersect(anotherHash)

Performs an intersection with the given hash. This is done by matching the keys of both hashes. Values of anotherHash are used. For example:

var morePopulations = new Data.Hash({'austria': 8402909});
populations.intersect(morePopulations).toJSON();
// => {"austria": 8402909}

union    hash.union(anotherHash)

Performs a union with the given hash.

var morePopulations = new Data.Hash({'spain': 46030109});
populations.union(morePopulations).toJSON();
// => {"austria": 8402908, "germany": 81742000, "usa": 310955497, "spain": 46030109}

Data.Graph

Now this is the most advanced data-structure offered by Data.js. A Data.Graph can be used for representing arbitrary complex object graphs and can be traversed in various ways. Relations between objects are expressed through links that point to referred objects. A Data.Graph consists of arbitrary many Data.Nodes. We distinguish between two different kinds of nodes:

  • Data.Objects describe entities that are member of certain types
  • Data.Types describe schema information

A Data.Graph works similar to the type system of an object oriented programming language. A Data.Type conforms to the concept of a class and defines abstract characteristics of a thing, the properties. Data.Objects then, are instances of such types. In contrast to most programming languages, Data.Objects can be member of multiple types, sharing their properties in a prototypal inheritance manner. An object's type property specifies an array of types, which corresponds to a prototype chain (cmp. Javascript). If there are properties with the same key on different types the latter wins. Data.Graphs can be manipulated by Data.Transformers that allow you to specify individual computations in order to generate a new graph based on an existing input graph.

Let's have a look at an illustrative example that outlines the basic manner of use. First off we formulate a little schema that will serve as our type system:

Usage

var schema = {
  "/type/person": {
    "type": "/type/type",
    "name": "Person",
    "properties": {
      "name": {"name": "Name", "unique": true, "type": "string", "required": true},
      "origin": {"name": "Origin", "unique": true, "type": "/type/location" }
    }
  },
  "/type/location": {
    "type": "/type/type",
    "name": "Location",
    "properties": {
      "name": { "name": "Name", "unique": true, "type": "string", "required": true },
      "citizens": {"name": "Citizens", "unique": false, "type": "/type/person"}
    }
  }
};

We are now ready to instantiate our very first Data.Graph, containing two Data.Types, but yet no Data.Objects.

var graph = new Data.Graph(schema);

Let's add some objects.

graph.set({
  _id: "/person/bart",
  type: "/type/person",
  name: "Bart Simpson"
});

graph.set({
  _id: "/location/springfield",
  name: "Springfield",
  type: "/type/location",
  citizens: ["/person/bart"]
});

Altough Springfield is aware of bart as a citizen, Bart doesn't have an origin yet.

graph.get('/person/bart')
  .set({origin: '/location/springfield'});

Well, now Homer wants to join the fun.

graph.set({
  _id: "/person/homer",
  type: "/type/person",
  name: "Homer Simpson",
  origin: "/location/springfield",
});

Mayor, there's a new citizen!

graph.get('/location/springfield').set({
  citizens: ['/person/bart', '/person/homer']
});

Now Mayor Quimby wants to display a list of citizens. Luckily he's got some basic Javascript skills:

graph.get('/location/springfield').get('citizens').each(function(person) {
  console.log(person.get('name'));
});

Eventually, this whole thing is all about creating applications with a dynamic type system, either in the browser or using Node.js. You can at any time adjust your types by adding or removing properties.

Persistence

Now best of all, Data.js 0.2.0 also adds a persistence layer that lets you synchronize your local Data.Graph with a database backend. When using Node.js, you might want to use the provided Data.CouchAdapter. In order to synchronize both, the schema and the objects, we need to mark our type nodes as dirty, so they get pushed to the DB as well. Therefore we merge in our schema again, this time marking the nodes as dirty. Then we're ready to start a sync. However, before all this we need to connect the graph to a data-store using a Data.Adapter.

graph.connect('couch', { url: 'http://localhost:5984/simpsons' });
graph.merge(schema,{dirty: true}); // nodes should be considered dirty.
graph.sync(function(err) { if (!err) console.log('Successfully synced'); });

That was easy. If another client wants to access that data the following steps need to be performed:

var graph = new Data.Graph(schema); // An empty graph
graph.connect('couch', { url: 'http://localhost:5984/simpsons' });

Once the graph is connected to a data-store we can query for data:

LocalStorage

When used in the browser you can make your Data.Graph persistent using LocalStorage.
var graph = new Data.Graph(schema, {persistent: true});

Once you reload the page, the graph gets restored from LocalStorage data. This is particularly useful for offline scenarios. If there's no internet connection your changes can be stored locally, ready to be synced as soon as your'e back online.

Methods

Now that you know the basics, here's the full API for reference.

set    graph.set(id, properties)

Set a new node on the graph. You can either choose your own id or let the system create an artificial guid for you (Data.guid([prefix])). Don't forget to specify a type property, depicting a list of types, your object is a member of. Order matters if there are property key collissions.

graph.set('/person/bart', {
  type: '/type/person',
  name: 'Bart Simpson'
});

get   graph.get(id)

Get node by id from the local graph space.

var bart = graph.get('/person/bart');
bart.get('name') // => 'Bart Simpson'

del   graph.del(id)

Delete node by id. Sets a _deleted: true on the node. In a persistence scenario the node gets removed from the database on the next sync(). Example: graph.del('/person/marge') 

merge    graph.merge(subGraph, dirty)

Merges in another Data.Graph. merge accepts a serialized subGraph, expressed as JSON. dirty indicates whether the corresponding nodes should be marked as dirty or not. Dirty nodes are persisted to the database on the next invocation of Data.Graph.sync().

graph.merge({"/type/person": {type: "/type/person", name: "Marge Simpson", location: "/location/springfield"}});

empty    graph.empty() 

Removes all objects from the graph. Type nodes remain.

find    graph.find(qry)

Find nodes in the graph that match a particular query object. As a result a Data.Hash containing the matched nodes is returned. See section Queries for more information about how to construct query objects.

graph.find({type: "/type/person", name: "Marge Simpson"});

fetch   graph.fetch(qry, [options], callback)

Fetch additional nodes from the server, asynchronously. You receive a callback when the data is ready. See the Queries  section for more information about how to construct query objects.

graph.fetch({"name": "Homer Simpson"}, function(err, nodes) {
  console.log(graph.get('/person/homer'));
});

sync   graph.sync(callback)

Synchronize your local graph with the database. Only dirty nodes are considered. You receive a callback when the sync has been finished. Keep in mind that nodes are validated before they are synced. Invalid nodes remain in your local graph until they get valid. You can check for invalid nodes by calling graph.invalidNodes(). Also there's the possibility of conflicts in a multi-user scenario. Like CouchDB, Data.js uses revisions to detect conflicts. After performing a sync you can look up possibly conflicted nodes by calling graph.conflictedNodes(). If nodes gets rejected for another reason (e.g. when a filter detects that your're not authorized for a certain operation), you can look them up with graph.rejectedNodes().

graph.sync(function(err) {
  if (!err) console.log("successfully synced");
});

// If you want your nodes to be synced on the fly you can bind a function to the dirty event.
// It's triggered every time the Data.Graph gets modified.
graph.bind('dirty', function() {
  graph.sync(function(err, invalidNodes) {...});
}

// You can as well listen for conflicts by binding a function to the 'conflicted' event.
graph.bind('conflicted', function() {
  alert('conflicts detected.');
});

// The same works for invalid, and rejected
graph.bind('invalid', handleInvalid);
graph.bind('rejected', handleRejected);

watch   graph.watch(name, query, callback)

Subscribe for graph updates in realtime. See usage at section "Realtime graph updates".

types   graph.types()

Returns only Data.Type nodes. Useful for schema inspection.

objects   graph.objects()

Returns all Data.Object nodes of the current graph.

dirtyNodes   graph.dirtyNodes()

Returns a Data.Hash containing dirty (changed, but not yet synced) nodes.

invalidNodes   graph.invalidNodes()

Data.Objects that haven't passed the obj.validate() test. You might want to inspect the object's errors property to learn more details.

graph.invalidNodes().each(function(node) {
  console.log(node.errors);
});

conflictedNodes   graph.conflictedNodes()

Returns a Data.Hash containing conflicted nodes. Conflicts can occur in multi-user scenarios when syncing with the database. Data.js uses node revisions (cmp. CouchDB) to detect conflicts. One way to resolve them is fetching the corresponding nodes again, apply the local updates and call Data.Graph.sync() again:

rejectedNodes   graph.rejectedNodes()

Returns a Data.Hash containing rejected nodes after a sync. Rejected nodes result from a filters that do not accept certain nodes to be saved.

Data.Type

A Data.Type denotes an IS A relationship about a Data.Object.  For example, if you type the object /person/shakespeare with the type /type/person , you are saying that Shakespeare IS A person. Types are also used to hold collections of properties that belong to a certain group of objects.

properties   type.properties()

Returns the properties of a type as a Data.Hash. Useful for schema inspection.

var person = graph.get('/type/person');
person.properties().each(function(p, key) {
  console.log(p.toJSON()); // Property inspection
});

objects   type.objects()

Returns Data.Objects associated with the type.

person.objects().keys(); // => ["/person/bart", "/person/homer"]

toJSON()   type.toJSON()

Serialize Data.Type as JSON for exchange.

person.toJSON();
// =>
// {
//  "type": "/type/type",
//  "name": "Person",
//  "properties": {
//    "name": {"name": "Name", "unique": true, "type": "string"},
//    "origin": {"name": "Page Count", "unique": true, "type": "/type/location" }
//  }

Data.Object

A Data.Object represents a data object within a Data.Graph.  It provides access to properties, defined on the corresponding Data.Types an object belongs to.

get  obj.get(property)

Returns for a given property key the corresponding value(s). Depending on the corresponding schema information, the right object is returned.

We distinguish four access scenarios:

  • Unique value types — obj.get('name') // => "Homer Simpson"
  • Non-unique values types —  obj.get('hobbies') // => ["Running", "Swimming", "Cycling"]
  • Unique object types — obj.get('location') // => /location/springfield
  • Non-unique object types — obj.get('citizens') // => [/person/homer, /person/bart]

set  obj.set(properties)

Sets properties on the object. Existing properties are overridden. Be aware that your property values must conform to the particular property's type specification.

var homer = graph.get('/person/homer');
homer.set({name: "Homer Jay Simpson", location: "/location/springfield"});

types   obj.types()

Returns all Data.Types the object is a member of. Usage: obj.types()

properties   obj.properties()

Returns Data.Properties from all associated types. Usage: obj.properties()

validate   obj.validate()

Validates an object against its schema (types)

var ned = graph.set('/person/ned', {
  location: '/location/springfield'
});

ned.validate() // => false
ned.errors // => [{property:'name', message: 'Property "name" is required'}]

toJSON   obj.toJSON()

Serialize Data.Object as JSON for exchange.

Data.Collection

A Collection is a simple data abstraction format where a data-set under investigation conforms to a collection of data items that describe all facets of the underlying data in a simple and universal way. You can think of a Collection as a table of data, except it provides precise information about the contained data (meta-data). A Data.Collection just wraps a Data.Graph internally, in order to simplify the interface, for cases where you do not have to deal with linked data. This is particular useful for data analysis and visualization of numeric (e.g. financial) data. Actually, data analysis and visualization were the primary use-case Data.js was built for.

Data.Collections can be exchanged in a simple JSON format:

Usage

var countries_data = {
  "properties": {
    "name": {"name": "Country Name", "type": "string", "unique": true },
    "official_language": {"name": "Official Language", "type": "string", "unique": true },
    "population": { "name": "Population", "type": "number", "unique": true }
  },
  "items": {
    "austria": {
      "name": "Austria",
      "official_language": "German",
      "population": 8356700,
    },
    "ger": {
      "name": "Germany",
      "official_language": "German",
      "population": 82062200,
    },
    "usa": {
      "name": "United States of America",
      "official_language": "English",
      "population": 310955497,
    }
  }
}

Once constructed, you can perform calculations on it.

var countries = new Data.Collection(countries_data);

var population = countries.properties().get('population');
population.aggregate(Data.Aggregators.SUM) // => Returns total population of all countries in the collection.

Say we are interested in the populations grouped by language. That can be done using the Data.Collection.group() method.

var languages = countries.group(["official_language"], {
  "population": {aggregator: Data.Aggregators.SUM, name: "Total Population"}
});

languages.at(0).get('name'); // => "German"
languages.at(0).get('population') // => 90418900
languages.at(1).get('population') // => 310955497 for "English"

Methods

The API of a Data.Collection is simple. It's composed of Data.Objects and one implicit Data.Type /type/item. That's because internally, it wraps a regular Data.Graph and just exposes a simplified interface.

get  collection.get(id)

Returns for a given id key the corresponding item from the collection.

set  collection.set(id, properties)

Sets (adds) a new item on the collection. Internally it delegates to Data.Graph.set(). For you as a user the only difference is, that you do not need to specify a type property.

countries.set("france", {
  name: "France",
  official_language: "French",
  population: 65821885
});

find  collection.find(query)

Find objects that match a particular query. See Data.Graph.find() for usage.

filter  collection.filter(query)

Returns a filtered collection containing only items that match a certain query. Works like Data.Collection.find(), but instead of a list of items, a whole new filtered Data.Collection is returned.

group  collection.group(groupKey, properties)

Returns a new grouped (transformed) collection. groupKey specifies the properties describing the group (e.g. ["official_language"]). By default, Data.Aggregators.SUM is used for aggregating numeric property values. Additional properties can be specified with the second parameters.

var languages = countries.group(["official_language"], {
  "population": {aggregator: Data.Aggregators.SUM, name: "Total Population"}
});

properties  collection.properties()

Return all properties associated with the collection.

items  collection.items()

Return all items associated with the collection.

Queries

Data.js allows you to query for data, either locally (Data.Graph.find()) or remote (Data.Graph.fetch()). Both methods take a query object that describes the properties that all returned objects must conform to. Query objects are basically key value pairs, but support a range of operators that are suffixed to property keys. The following operators are supported for local queries and dynamic remote queries:

  • == — Equal (default)
  • != — Not Equal
  • > — Greater Than
  • >= — Greater Than Equal
  • < — Lower Than
  • <= — Lower Than Equal
  • |= — Any Of
  • &= — All Of
Example:
var filteredCountries = c.find({
  "languages_spoken&=": ["English", "French"],
  "form_of_government|=": ["Constitution", "Democracy"],
  "population>=": 70000000
});

Indexed Remote Queries

With Data.js 0.3.0 fast indexed queries are used by default. In order to make this work you need to specify indexes on your schema nodes. If you need to lookup persons by name and origin your type definition would look like so:

"/type/person": {
    "type": "type",
    "name": "Person",
    "properties": {
      "name": {"name": "Name", "unique": true, "type": "string", "required: "true"},
      "origin": {"name": "Origin", "unique": true, "type": "/type/location" }
    },
    "indexes": {
      "by_name_and_origin": ["name", "origin"]
    }
  }

Indexes are created automatically when your type node gets saved. You can then utilize the following indexed query.

var qry = {type: "/type/person", "name": "Homer Simpson", "origin": "/location/springfield"};
graph.fetch(qry, function(err, nodes) {
  // matched nodes from the the database
});

Please note that remote queries do not support operators, as they're not compatible with database indexes. However for cases where a query doesn't match a registered index, objects in the database are matched against the query dynamically. Please note that this tends to be slow, and should be avoided.

Include associated objects

With 0.4.0 the query interface got a notable improvement. Users can now specify placeholders in order to fetch objects along with associated (referenced) objects.
{
  "_id": "/person/homer",
  "children": {
    "_recursive": true
  }
}

The above query tells the query execution engine that Homer's children should also be part of the result. For Homer these are /person/bart, /person/lisa, /person/maggie. This conforms to the concept of eager loading as you need just one single request to reach those associated objects. With the _recursive directive you can even include children's children. Supposing that Bart or Lisa will ever have children, they would be fetched too. Substance uses this technique to fetch a document along with all associated content nodes (Sections, Sub-sections, Text, Images etc.).

Realtime graph updates

As of Data.js 0.3.0 clients can watch for graph updates in realtime. Here's what you need to in order to get started.

The server

var Data = require('data');

// Init a graph with your schema (type nodes)
var graph = new Data.Graph(schema, false);

// Connect to a datastore
graph.connect('couch', { url: "http://127.0.0.1:5984/simpsons"});

// Serve the graph with an HTTP Express.js server instance
graph.serve(server);

After calling graph.serve the server provides both, an AJAX Endpoint and a WebSocket server, if you choose to use a realtime transport for synchronizing your graph. The client can then choose the transport type that should be used. We're going to go realtime, with the nowjs adapter.

The client

First include the client files, which are served by graph.serve(). 

<script type="text/javascript" src="/nowjs/now.js"/>
<script type="text/javascript" src="/datajs/data.js"/>

Setting up the client is easy:

var graph = new Data.Graph(schema).connect('nowjs');

// We need to wait a bit until the Websocket connection has been established
graph.connected(function() {
  // Now let's watch for external graph updates
  graph.watch('persons', {"location": "/location/springfield"}, function(err, nodes) {
    // Called each time persons originating in Springfield are updated.
    // Make your UI updates here
  });
});

Please also have a look at a full example, showcasing realtime functionality.

Filters

By default there's no layer of security preventing from unauthorized graph access. While this is handy during development, in production you want to secure your graph's read and write operations. Filters function as a middleware and let you hook into graph reads and writes. A filter stack is applied to a Data.Graph like so: 

Usage

Filters = {};
Filters.ensureAuthorized = function() {
  return {
    read: function(node, next, session) {
      if (authorized(node)) {
        delete node.email; // hide email address
        next(node); // node is accepted
      } else {
        next(null); // reject node
      }
    },
    write: function(node, next, session) {
      next(null); // disabled write operations completely
    }
  }
};

// Apply the filter(s)
graph.connect('couch', {
  url: "http://localhost:5984/simpsons",
  filters: [Filters.ensureAuthorized()]
});

// Some graph manipulation
graph.sync() // is now secure.

The filter stack is applied for each node involved in a read or write operation. In a filter you can accept or reject nodes, but also manipulate them. Filters are rather low level and operate on raw javascript objects. However you can still fetch additional data by calling this.db.get('nodeid') or this.read(qry, fn) from within your filters.

Best Practices

Have your schema at hands

On both sides, the server and the client, it's good practice to have the schema statically bootstrapped. By doing so you can save an additional request to the database that would be necessary to fetch the implied type nodes.

In a Node.js app you might want to keep your schema in db/schema.json ready to be read and cached when the server starts and also to be served with each initial page request, so the client can construct an empty Data.Graph right away.

Just for the sake of completeness: There may be applications where you want your schema to be dynamic in the sense of allowing users to change not only objects, but also modify type nodes during run-time. In such a case you wouldn't use the described static approach.

Use one graph per request

When used on the server-side you should be aware that a Data.Graph potentially grows fast. So if you are thinking about using just one global graph instance for the whole app, do not. You'll run out of memory if you don't care about removing objects manually. A better strategy is to use one graph instance per request. Data.Graphs are lightweight after they've been constructed with just the application's domain schema. You can then operate on that local graph instance (using fetch, sync as usual) while being assured that memory is being released when the request has been finished. You can have a look at the Substance Source Code for a real world example. 

Schema Evolution

With Data.js there's no big difference between schema and data. Type nodes are just regular nodes that have the type property set to /type/type. However, every time a type node gets updated, index-views are rebuilt and validation functions are updated. Like with objects, a valid revision (_rev) needs to be supplied when trying to update an existing type node. Since we'll maintain our schema in a static file that's not exactly handy. For the purpose of schema evolution we rather want to force updates. In our projects (Substance, Dejavis) we use a small Seed Script to update our schema after we've modified the applications domain schema. You can either perform a non-destructive upgrade ($ node db/seed.js) or reset the whole database ($ node db/seed.js --flush).

Background

Based on the Metaweb Object model

The Data.Graph format is highly inspired by the Metaweb Object Model that is used at Freebase.com. So if you're familiar with Freebase and MQL, you should have already gotten the basic idea. 

Why not RDF?

Actually, we were considering building this framework on top of an existing RDF-based serialization format. However we ended up with introducing our own JSON based format for various reasons:

  • Javascript implies JSON ;-)

  • RDF is designed to work in a global distributed scenario, involving a more verbose syntax. A Data.Graph operates in a local scenario, and therefore allows for a tighter syntax.

  • From our experience, proprietary formats are perfectly valid as long as mapping data back and forth is easy. Since RDF and Data.Graphs are both modeling a graph, translation should be easy enough.

  • Standardized Ontologies are important for the Semantic Web, but for the task of client-side data-processing they are most often irrelevant.

However, in the future RDF support (for construction and serialization) may be added to the library. Until then, a scenario involving RDF could look like so:

  1. Fetch data from a SPARQL endpoint
  2. Translate the result to the Data.Graph format
  3. Do data processing using Data.js
  4. Display the results on the fly (e.g. using a visualization for encoding the results)

Examples

Tasks

Tasks is meant to be used as a starting point for building applications on top of Data.js. Feel free to contribute!

Substance

Substance is a web-based document authoring and publishing platform. It uses Data.js for all data-related concerns. Those are client-side data-manipulation like faceted filtering as well as data persistence with CouchDB.

Déjàvis

Déjàvis is a tool for analyzing and visualizing data. Data manipulation (like filtering, grouping and aggregating) is done through Data.js

Release Notes

0.4.1 — July 24th 2011

  • Adds experimental support for LocalStorage (Data.Graph snapshots)

0.4.0 — July 15th 2011

  • Query Engine now supports eager loading of associated objects
  • Validations are additionally stored as CouchDB validation functions, thus ensured on DB-level
  • Query operator defaults to |= when an array is passed as a value
  • Data.Graph#set now takes just one parameter (_id is specified as a property)
  • Added Data.Graph#empty
  • Simplified interface for Data.Graph#sync
  • Unit tests for the persistence API

0.3.1 — May 28th 2011

  • Fixed a bug in Data.Graph#merge
  • Made Data.Hash#intersect and Data.Hash.difference actually fast
  • Only consider own properties in Data.Hash.
  • Enable derived properties in group operations
  • Deal with the special case where a collection has zero properties

0.3.0 — Apr 27th 2011

  • Indexed queries against CouchDB
  • Completely rewritten Couch Adapter
  • Watch for external graph updates in realtime Data.Graph#watch
  • Data.Graph#find now accepts an array of queries
  • Added support for Filters that let you hook into graph reads and writes
  • Re-implementation of value bookkeeping (registration, unregistration)

0.2.2 — Mar 27th 2011

  • Changed Data.Graph.fetch() signature
    • options are now optional
    • returns a Data.Hash of fetched nodes
  • Data.Graph.merge() is now chainable
  • Added Data.Hash.difference()
  • Added Data.Hash.rest()

0.2.1 — Mar 12th 2011

  • Improved query interface
  • Added Data.Collection.find()
  • Added Data.Collection.filter()
  • Added ALL-OF Operator (&=)
  • Renamed ONE-OF to ANY-OF (|=)
  • ANY-OF operator now also works with unique properties
  • Now using deep equality check for comparing property values

0.2.0 — Mar 10th 2011

Added a Graph Persistence Layer to store graphs, an adapter interface for connecting data-stores, and a concrete implementation that uses CouchDB in the backend.

0.1.0 — Nov 28th 2010

Initial release.