Using Protocol Buffers#

Query format#

It is possible to serialize queries using Protocol Buffers in binary form, instead of sending it as a text in edn or GraphQL format.

Hiku has a hiku/protobuf/query.proto file, which describes message types for query serialization.

Here is how they can be used to build query:

from hiku.protobuf import query_pb2

node = query_pb2.Node()

link = node.items.add().link
link.name = 'characters'

field = link.node.items.add().field
field.name = 'name'

This query is equivalent to this query in edn format:

[{:characters [:name]}]

And is equivalent to this query in GraphQL format:

{
  characters {
    name
  }
}

Note

Protocol Buffers has it’s own ability to specify requirements for get operations – google.protobuf.FieldMask, but this approach has limitations for our use. For example, field options can’t be expressed with field masks. So it would be hard to utilize field masks for our use case and that’s why Hiku provides it’s own message types for queries.

Query export#

In Python it is not required to use example above in order to build query using Protocol Buffers message classes. Hiku provides handy export() function to transform query into Protocol Buffers message:

from hiku.builder import build, Q
from hiku.export.protobuf import export

query = build([
    Q.characters[
        Q.name,
    ],
])

message = export(query)
assert message == node

binary_message = message.SerializeToString()

Query reading#

In order to execute query, Hiku provides read() function, which can be used to deserialize query from Protocol Buffers message:

from hiku.readers.protobuf import read

query = read(binary_message)

result = hiku_engine.execute(graph, query)

Result serialization#

The main advantage of using Hiku with Protocol Buffers is to give efficient binary format for result serialization, which can be safely read in any other language with Protocol Buffers support.

Note

Hiku is only suitable with latest 3rd version of Protocol Buffers format. This is because only in 3rd version of Protocol Buffers all message fields are strictly optional, and this opens possibility for clients to express their requirements and server will return only what client cares about.

Here is our example of a graph, similar to the one from Basics:

GRAPH = Graph([
    Node('Character', [
        Field('name', String, character_data),
        Field('species', String, character_data),
    ]),
    Root([
        Link('characters', Sequence[TypeRef['Character']],
             to_characters_link, requires=None),
    ]),
])

This graph can be expressed like this in example.proto file:

syntax = "proto3";

message Character {
  string name = 1;
  string species = 2;
}

message Root {
  repeated Character characters = 1;
}

Now we can generate example_pb2.py file from example.proto by using protoc compiler. And this is how it can be used to serialize result:

In this example server will compute only name field of the Character type, because this is the only field, which was specified in the query.

Note

Using Protocol Buffers “as is” still not the most efficient way to send result back to the client. Result is sent in denormalized form, so it contains duplicates of the same data. More efficient way would be more complicated and probably will be implemented in the future. See Netflix/Falcor and Om Next as examples of using normalized results. See also how Hiku stores result internally: hiku.result