Argot meets Contiki (Part 2)

Back in October last year I wrote about building an example of using Argot on Contiki. After releasing Argot recently under a BSD licence, I've been working on preparing that small project for release. I'm happy to say that release is now done. It builds on Argot, Colony and the newly released Argot MCG (Micro Code Generator) which generates C code from Argot dictionary files. You can find all the project files at www.einet.com.au/Downloads.

At a high level the Contiki Argot project shows embedding a protocol meta data description in a small Contiki device. The meta data is used by a client to verify the definition of a protocol before sending a request to the device. Placing Argot meta data on the device allows each part of an interface to be versioned and allows clients to perform full protocol discovery on devices with only a few kb of read only memory. On the client side, a developer is able to use java interfaces which bind to the protocol description allowing fast development of clients. This form of protocol discovery allows tight binding between client and server while still maintaining loose coupling. For details of what the example shows you can read the details in the post from October.

If you're interested in trying out the example, I suggest downloading the instant contiki vmware development environment. From a command shell in Contiki get the project:

wget http://www.einet.com.au/files/ContikiArgot-1.0.0.zip
unzip argot-contiki-1.0.0.zip
cd argot-contiki

To build the server a small change needs to be made to the Contiki build system (applies to contiki-2.4). The following command will patch the makefile with the required changes:

patch -d /home/user/contiki-2.4

In these instructions the server is being built to be run using the minimal-net configuration which is run on the unix host. This is useful for debugging and as a demonstration of the code. The Contiki minimal-net configuration uses the tap0 unix networking driver to put the host into a different network. This is configured in the “contiki-2.4/platform/minimal/contiki-main.c” file. I use ip 192.168.22.3, netmask 255.255.255.0, gateway 192.168.22.1. In the “contiki-2.4/cpu/native/net/tapdev.c” the tap0 configuration needs to be adjusted for your selected IP address. Around line 107, change the line “ifconfig tap0 inet 192.168.22.1” to match your configuration.

Following configuring the network components of the example you can build the server:

cd examples/test
make

To run the server:

sudo ./argot-server.minimal-net

Leave the server running and open a different shell. To build the Java client code, two jar files need to be copied into the Apache Ant build libraries. This is a one off task:

cp lib/junit.jar /usr/share/ant/lib
cp lib/ant-junit.jar /usr/share/ant/lib

You can then build the java client by calling ant at the base of the project:

ant

After the client has been successfully built, you can call the simple test code:

java -jar lib/contiki-argot-1.0.0.jar 192.168.22.3

If everything is successful you should have an output that ends with “doSomething returned: 30”.

Obviously, this example is still quite rudimentary. However, it does demonstrate calling a Contiki based service which uses only a few kb of memory. It also shows a client which only requires a Java Interface to be coded to make the call. Next steps are to improve the functionality of the test and look further into the transport protocols used in this environment.

Argot, Patents be gone, BSD free

The concepts behind Argot has been under consideration for a patent over the last six years. After some deliberation I have decided to abandon the patent and change the way Argot is released. In the coming weeks the Argot source code will be released under a BSD style license. This completely removes the shackles of patents and licensing which I believe may have been holding Argot back. This is a major change to the way Argot is released and will hopefully pave the way for it to grow and improve.

A major motivation for removing the patent and license is timing. The M2M (Machine to Machine) market is evolving and growing quickly at the moment. The concept of the Internet of Things (IOT) is becoming the next big thing and projects like Contiki are gaining momentum. For Argot to have any chance of being accepted, it needs a simple compatible license. Argot also needs to be tested in the market and possibly evolve to the meet the needs of the market. It can't do that sitting on the shelf. Argot is designed to meet the needs of the embedded and M2M market; small light weight data encoding with data type versioning built-in. No other technology can provide the ability to embed the full meta description of a protocol in as little as 4kb of memory. This is the right time for Argot to be in the market and tested fully.

If you're a developer or technologist of any sort that had been turned off Argot in the past, because of the wrong license or patents, please, look again safe in the knowledge that the code is free from restrictions. In fact, one of the consolations of having an abandoned patent is that you know that its safe from other patents getting in the way in the future; prior art is proven back to 2003. If you like what you see, or think it's climbing the right mountain please get involved or even develop your own implementation.

Wedding!

As this blog is generally focussed on my thoughts on distributed computing and my ongoing advances with Argot, it is unusual that I would delve into personal elements of my life. However, this post will be a little different, as a month ago my long time partner Tracii and I got married. Both of us were so impressed with how the day went that I felt it necessary to record a few of the memories and provide some links to the many wonderful businesses and services that made our day extra special.

Obviously, the elements of a wedding don't come together by themselves. While I like to think I put my fair share of the effort into the planning and organisation, I must thank my wonderful wife for the real heavy lifting. She kept track of when everything had to be done and did a great job bringing together venue, car, band, flowers, celebrant, photographer & invitations. She also brought together the colour scheme for the day, and her success in doing so can be seen in our wonderful photos.



We got married at Stones of the Yarra Valley in Coldstream, Victoria. This is not the cheapest venue in town, but wow, what a stunning location to get married and have a party. Steve and Vonnie Frazer who own and run the place have obviously done plenty of weddings in their time. They were very patient and happily answered all the small questions that we had, and their staff made sure everything ran to schedule on the day.



The food and wine provided at the reception was delicious and our guests all gave us lots of lovely feedback. Our only negative issue was that the seating arrangements we had organised somehow got lost in translation on the day, which meant that fifty percent of our guests were not where we had expected them to be. Regardless of this, everyone had a fabulous time.

A really important aspect of any wedding day is obviously the photography. I'd probably say that having a photographer whose work you like and that you have some rapport with is among the most important things to get right. We used my cousin Lauren Marshall, whose work is fantastic for someone who is just starting out with wedding photography. All the photos displayed here are by her and her assistant Rebecca Humphries, and clearly show how great their work is.



A surprise stand out service of our day was the chauffeur and car that we hired through Art Deco Wedding Cars. The two tone convertible 1951 Mark V Jaguar is obviously a beautiful car. However, Guy Gleeson who owns and runs the business did an amazing job on the day. He provided umbrellas to stop us from squinting in our photos, lollies to get kids attention during the family portraits, and sprayed water on Tracii's bouquet to freshen it up directly before she entered the chapel. He also allowed us and various kids (and even some adults!) to have photos taken in and around the car.



Another stand out for Tracii and I were the band. Music has such an important role in setting the mood for the reception and was a highly important element for both of us. One of Tracii's favourite Australian bands is The Blackeyed Susans who have been together for twenty years. When Tracii first suggested hiring them to play we honestly didn't think we'd have a chance at getting them to agree, however they were actually surprisingly enthusiastic and well priced.



Although we weren't able to get the full band as the drummer and double bassist were on holidays, Rob Snarski and the others did a superb job. Having an original band play their music made the night even more memorable.



Another important element of the wedding was the design. A good designer can create a theme which is carried through invitations, place cards, guest book & thank-you cards. I'm very lucky to have a wonderful sister Joanne, who has been doing graphic design for over twenty years through her company Socket to Me. She took our thoughts and created products that exceeded our expectations. During the creative process, Tracii and I were able to heavily influence the design, with Jo using her talents to bring our ideas to fruition.





A major part of any wedding are the outfits of the bridal party. Tracii's dress and bridal jewellery came from Ravish at Knox City, in Wantirna South. She managed to find a dress which fitted her off the rack and looked exquisite. We were very fortunate to have my Mum pay for Tracii's wedding dress, do all the necessary hem alterations and make the veil.



Tracii's son Dane wore a suit hired from Prinzi Collections in Carlton. Dane's silk tie and handkerchief were also created by my Mum, as was the handkerchief in my suit.

Tracii and I scoured the city for the right suit for me. After looking through heaps of places we found Rick at Soho Workshop in Little Collins St, Melbourne who went out of his way to meet our needs. After listening to what we were after, Rick suggested that he had a suit at his other store that would work beautifully with our colour scheme. Not only was the suit the perfect fit and colour, but he was also able to provide us with a one-off matching waistcoat that another customer had custom made, but changed their mind about. By some miracle, this waistcoat was an almost perfect fit! Soho Workshop also supplied the shoes, belt, shirt and tie and tailored everything to fit in the week before the wedding.



Detail was added to the wedding through the floral arrangements in the bouquet, reception, chapel and button hole flowers. We were lucky enough to have a friend who just purchased a florist business. Michelle Dann runs Alchester Village Florist in Boronia, and did a wonderful job with every element. She was given a basic idea of colour scheme and which flowers to use and managed to create arrangements which we were both very impressed with.





Tracii's hair and make up was done by Debra Atkin who also recently purchased a business, Colour Me In Hairdressing in Kallista, who also did a fantastic job.



The wedding cake was supplied by Fantasy Cakes in Doncaster , who were given a design and description by us and delivered a gorgeous chocolate mud cake on the day, which everyone agreed tasted divine.



Our celebrant Maree Livy helped us bring together a service which reflected exactly what we wanted. Between the three of us we created a ceremony which we were happy with, and on the day she delivered it accurately and didn't incorporate any surprises. Her tone reflected the happiness and joy of the occasion perfectly.



The bridal party and a lot of our guests chose to stay in the Yarra Valley after the wedding at the beautiful Yering Gorge cottages. The staff organised a 24 seater bus to take our guests to and from the wedding, as well as a delicious buffet brunch on the following day. The wonderful environment at Yering Gorge certainly added to the relaxed feel of the wedding experience.

Special mention must also go to Tracii's friend Dr Tim Byron, who flew down from the Gold Coast to play a selection of Sigur Ros songs on the grand piano during the ceremony.

Other suppliers who we used and recommend are:

G&N Jewellers who made our custom wedding rings, Novat Shoes who custom made Tracii's wedding shoes, Natskin Balgownie who provided facials, manicures, pedicures for the bride and massages for both of us on the morning of the wedding, Paper Indulgence who supplied our stationary for our invitations and place cards, as well as the guestbook, Pink Noise Audio who provided the PA equipment and sound engineer for the band, as well as taking care of the iPod playlists we provided them with, and Kudos Villas in Daylesford whose luxurious accommodation ensured that we had the most relaxing and romantic honeymoon imaginable.

At the end of the day, it was the sharing of a special moment in mine and Tracii's life with our friends and family which was the most important of all. Overall it was a fantastic and hectic day that will never be forgotten.

Trillions - Climbing the right mountain

If you haven't already seen it, watch this great clip by the maya design group. It describes the problems that we are about to face with integrating the Internet of Things in the most succinct way I've seen to date.

Trillions from MAYAnMAYA on Vimeo.

A section of it resonated with me greatly; that is that we are not going to create a network of Trillions of devices using the technology we have today. We need to climb a new mountain of ideas that will enable Trillions of devices to work together.

Looking a little further past the clip, you can find the White Paper by Peter Lucus, a founder of the Maya Design group on the challenges faced by creating a Trillion node network. An interesting point he makes is that in terms of information exchange, there have only been two universally accepted units of data representation; the bit and the byte. It has been forty years and there has been no improvement on this. The White Paper also discusses some of the architectural requirements that any solution must meet to make the Trillon node network a reality; these include scalability, tractability and comprehensiveness.

Argot/XPL is our attempt at climbing the right mountain. Argot is designed around those architectural requirements identified by the Maya Design group, however, at a more practical level I believe the solution must meet a number of other technical requirements. These are:

1. Strict Internal Data Models – Each node/file in a Trillion node network must have an internal consistent data model of the types of data it contains or it can communicate. This data model describes the structure of each element it can communicate. Most distributed systems already have a strict data model; however the data model is often referenced and is not specific to the node of the network. XML Schema offers a shared data model which makes it difficult to split schemas and implement parts of a schema. A Strict Internal Data Model describes the data types that the individual node is capable of communicating.



2. Element Versioning – The Data Model must be versioned at each individual element or structure. This is one area where XML Schema and most other data modelling tools currently fail at providing the necessary functionality to meet the needs of a Trillion node network. As functionality improves and changes, the data model must be updated. XML Schema only allows very limited changes which retain backward compatibility. Sooner or later an XML Schema requires changes which break backward compatibility. When backward compatibility is at this stage all nodes in the network must be updated to support new Schemas. This already happens in large corporate systems that use XML Schemas heavily.



3. Partial Comparison – It must be possible to perform a partial comparison of the data model. The data model of a Trillion node network is likely to be made up of many different schemas. Devices will implement different amounts of functionality from different shared data models. It must be possible to allow two nodes to directly discover if devices can communicate and discover the method of communications.

An important aspect of these requirements for building a Trillion node network of heterogeneous devices is that agreement on the data format is not a core requirement. The format could be Argot, or any other suitable format supporting those requirements. The requirements for the Trillion node network is on the data models and the nodes in the network to be able to allow change through versioning, partial data model implementation, partial data model comparisons and finally discovery mechanisms to allow agreement on data formats to be established.

Peter Lucus finishes his white paper by suggesting that attribute/value pairs might be the next universally agreed unit of data representation. This is where I disagree, the next unit of universal data representation needs to be the data model. From this point of view, data modelling to support change is very immature. I've seen very little practical implementations that deal with change acceptably in a heterogeneous network.

In the end, I agree with the Maya Group. We need to start climbing the right mountain to create suitable solutions to the Internet of Things. I don't believe we will get there using REST and XML as they don't support change, and change is one of the few requirements I can truly be certain is required as the Internet of Things become a reality.

Argot submitted to IETF

A little while ago Esmond Pitt and I submitted the “Extensible Presentation Language (XPL) and Type Resolution Protocol (XPL/TRP)” to the IETF as part of the 6lowapp working group. XPL is the name we have selected, however, the technology is based on Argot which has been developed over the past six years. I'm incredibly pleased with the result, as the document now provides the most clear description of what Argot is all about. The document draft-ryanpitt-6lowapp-xpl-00 is published on the IETF web site.

The following exerts from the proposal provide a nice short and concise description of the XPL/Argot concept:

“XPL was created to combine (i) the extensibility provided by XML and XML Schema, (ii) the ability to describe and encode binary information like IDL+IOP and ASN.1, (iii) the ability to create remote procedure call services like CORBA, and (iv) to be inherently 'version-aware', to ensure that a change to one aspect of the system did not create cost across the whole network.

The XPL solution is to have each application in the network contain metadata about all the information it can communicate (send and receive). This allows each XPL application to negotiate directly with other peers the information they are able to exchange. It is this fundamental change of moving the metadata knowledge into the application or file, rather than keeping it externally, which is the most important aspect of XPL.

The XPL type system embedded (or notionally embedded) in each device is combined with methods for performing type comparisons with external systems. The type comparisons are designed to find the common definition type set shared by two XPL type systems.”

The proposal breaks the technology into two parts. The following descriptions are also from the proposal:

1. XPL, a powerful Extensible Presentation Language which is used both at protocol design time and by applications at runtime via very simple and small code libraries to communicate both primitives and compound types. XPL creates an internally consistent directed graph, which may be cyclic, to define complete versioned type systems to be stored with (or notionally with) an application or device.
2. XPL/TRP, a compact Type Resolution Protocol, in turn provides (i) dynamic negotiation of protocols and their constituent types between disparate devices; (ii) protocol versioning all the way down to the type level; and (iii) dynamic discovery of device capabilities and versions. An XPL device is able to describe and communicate its own application protocol. All these features are built-in to the protocol and intrinsic to its operation, rather than being extra-cost additions to it.

The XPL system is a departure from how most application protocols are designed. Protocols are normally designed and then the implementation is created from the design. XPL binds the design and implementation together so that they are interrelated. This has interesting consequences for versioning and detailed discovery. A device can be interrogated to discovery the structure of all the data that can be sent/received to it. Combined with scripting languages and other methods it would allow clients to be built automatically with zero code. XPL is the first time that you can implant a formal protocol description in any device or application down to the smallest of devices.

There's a lot of other interesting proposals that have been delivered to the IETF. In the application protocols area includes, XPL, Binary HTTP (chopan), Binary XML (exi), ZigBee Alliance requirements, and others have been submitted. The area of M2M (Machine 2 Machine) is going to continue to hot up over the next few years. There's still plenty more work to do, and it will take a while before a consensus is reached over the various protocols. Some interesting times are ahead!

Argot Meets Contiki

I've spent the last week getting a demo Argot application communicating with a Contiki device. Thankfully, I was able to dust off the code developed in 2005 and get it working reasonably painlessly. The end result is a java client and a Contiki host with a published method that can be called via UDP from a Java application.

The Contiki device has implemented a very simple interface as defined by the following Argot source file. It defines a “test” interface and a “doSomething” method which is attached to the “test” interface. The “doSomething” method takes a single 32-bit integer as a request and responds with a 32-bit integer. The method multiplies the input by three and sends back the result.

(library.entry
 (library.definition meta.name:"test" meta.version:"1.3")
 (remote.interface))

(library.entry
 (library.relation #test meta.version:"1.3" u8utf8:"doSomething")
 (remote.method u8ascii:"doSomething"
  [ (remote.parameter #int32 u8ascii:"param" ) ]
  [ (remote.parameter #int32 u8ascii:"ret" ) ]
  [  ]
 )
)

This Argot source file includes types like “remote.method” and “remote.interface” that were not defined by the meta dictionary in the last post. These other types are also defined in other argot files. For brevity, I haven't included them here. I might cover them in a post later.

The java client uses the meta information supplied by the Argot dictionary to allow a Java Interface to be bound to the meta data. The result is that calling the method is simple:

public void testApp() throws Exception
{
 // Create the object reference
 MetaObject objectReference = new MetaObject(
   new SimpleLocation(2,"local"),
   _client.getTypeLibrary().getTypeId(ITestClass.TYPENAME,"1.3") );

 // Call the test method and check the result.
 ITestClass test = (ITestClass) _client.getFront( objectReference );
  
 // Call the remote method.
 int result = test.doSomething(10);
  
 assertEquals( result, 30 );
}

The first line creates the object reference. It consists of a location and a type. The location is an abstract data type meaning that different types of location specifiers can be used. In this case the location specifier is a simple Integer. The server contains a list of index based objects. For small devices this is appropriate, however, other devices may wish to use URLs or other location specifier. The second parameter is the identifier of the “test” interface.

The second line retrieves the interface to the “test” interface. In Java this has been implemented as the ITestClass. During setup the ITestClass is bound to the “Test” interface definition. All the methods are checked to ensure they are consistent with the methods that are defined in the Argot dictionary.

Finally, the method “doSomething” is called with the value 10. The end of the test asserts that the final result is 30.

On the Contiki side, I created a static dictionary structure of all the data types required by the interface. This included the full meta dictionary as defined in the last post, the “test” interface and method definition and everything in between. The end result is a dictionary with 60 entries and uses about 3kb of data. This can probably be halved with a few changes; I'll get to those later.

Argot uses a Type Resolution Protocol to create a tight data binding between client and server. The protocol uses only a few basic concepts. These are:

• TYPE_META – The initial request sends a challenge to request the meta dictionary. As this is the base for all other types, the client and server must ensure these are the same on both sides.


• TYPE_MAP – A type map request sends a data type dictionary location and definition and returns a type identifier.


• TYPE_MAP_DEFAULT – Sends a type name and returns a location with version and definition. The client then checks its own dictionary for a match.


• TYPE_RESERVE – In cases where the data type is self referencing the client must retrieve and identifier for the type to adequately define the type. This reserves an identifier before TYPE_MAP is called.


• TYPE_REVERSE – In cases where the server sends a client an identifier that the client hasn't mapped, the client sends the identifier and is returned the location with version and definition of the data type.


• TYPE_BASE – This is a type of boot strap discovery mechanism. The client is able to request the base type or interface to be found on the host.<.li>
  
  
• TYPE_MSG – Finally, the last message is a user defined message and is for the application protocol being used.

These basic messages allow a client to discover from first principles all the data types required to communicate with a host. The following demonstrates the messages being sent/received between the java application and Contiki host.

Note: The very first message should be a TYPE_META message. This is yet to be implemented on Contiki as it needs to be changed from the previous method of making this call. Previously the client would send the full meta dictionary so that the server can perform a binary compare. To reduce size this will be reversed. The client will issue a meta dictionary challenge; the server will respond with either the full meta dictionary or just a meta dictionary version. Obviously on small devices the ability to remove the 35 types from the meta dictionary will help size greatly. To reduce size further the response to the meta dictionary challenge could be a URL of where to find the device dictionary. In this way the benefits of the Argot protocol could fit the smallest of devices.

The following shows the full conversation between Java client and Contiki host:

MAP DEFAULT: remote
-> 02 00 09 0b 00 06 r  e  m  o  t  e 
<- 02 00 00 00 23 0b 00 06 r  e  m  o  t  e  00 01 05   

MAP DEFAULT: rpc 
-> 02 00 06 0b 23 03 r  p  c 
<- 02 00 00 00 3d 0b 23 03 r  p  c  00 01 05   

MAP DEFAULT: remote.rpc.request 
-> 02 00 0a 0b 3d 07 r  e  q  u  e  s  t 
<- 02 00 00 00 3e 0c 3d 07 r  e  q  u  e  s  t  01 03 00 23 0f 03 0e 08 l  o  c  a  t  i  o  n   0d 25 0e 06 m  e  t  h  o  d  0d 28 0e 04 d  a  t  a  10 0d 01 0d B    

MAP DEFAULT: remote.location 
-> 02 00 0b 0b 23 08 l  o  c  a  t  i  o  n 
<- 02 00 00 00 25 0c 23 08 l  o  c  a  t  i  o  n  01 03 00 02 07 00   

MAP DEFAULT: uint16 
-> 02 00 09 0b 00 06 u  i  n  t  31 36
<- 02 00 00 00 28 0c 00 06 u  i  n  t  31 36 01 03 00 09 13 10 10 04 16 10 17 18 19   

MAP DEFAULT: meta.identified 
-> 02 00 0d 0b 03 0a i  d  e  n  t  i  f  i  e  d 
<- 02 00 00 00 B  0c 03 0a i  d  e  n  t  i  f  i  e  d  01 03 00 11 0f 01 0e 0b d  e  s  c  r  i  p  t  i  o  n  0d 08   

MAP DEFAULT: remote.rpc.response 
-> 02 00 0b 0b 3d 08 r  e  s  p  o  n  s  e 
<- 02 00 00 00 3f 0c 3d 08 r  e  s  p  o  n  s  e  01 03 00 18 0f 02 0e 07 i  n  E  r  r  o  r  0d 3c 0e 04 d  a  t  a  10 0d 01 0d B    

MAP DEFAULT: bool 
-> 02 00 07 0b 00 04 b  o  o  l 
<- 02 00 00 00 3c 0c 00 04 b  o  o  l  01 03 00 02 0d 01   

MAP DEFAULT: index (definition) 
-> 02 00 08 0b 00 05 i  n  d  e  x 
<- 02 00 00 00 27 0c 00 05 i  n  d  e  x  01 03 00 04 0f 01 0d 28   

MAP: index (relation) 
-> 01 00 08 0d 25 05 i  n  d  e  x  00 02 06 27
<- 01 00 00 00 26   

MAP DEFAULT: remote.method 
-> 02 00 09 0b 23 06 m  e  t  h  o  d 
<- 02 00 00 00 2e 0c 23 06 m  e  t  h  o  d  01 03 00 33 0f 04 0e 04 n  a  m  e  0d 2f 0e 07 r  e  q  u  e  s  t  10 0d 01 0d 30 0e 08 r  e  s  p  o  n  s  e  10 0d 01 0d 30 0e 05 e  r  r  o  r  10 0d 01 0d 28   

MAP DEFAULT: u8ascii 
-> 02 00 0a 0b 00 07 u  38 a  s  c  i  i 
<- 02 00 00 00 2f 0c 00 07 u  38 a  s  c  i  i  01 03 00 10 12 10 0d 01 0d 01 09 I  S  O  36 34 36 2d U  S    

MAP DEFAULT: remote.parameter 
-> 02 00 0c 0b 23 09 p  a  r  a  m  e  t  e  r 
<- 02 00 00 00 30 0c 23 09 p  a  r  a  m  e  t  e  r  01 03 00 12 0f 02 0e 04 t  y  p  e  0d 28 0e 04 n  a  m  e  0d 2f   

MAP: remote.method (relation) 
-> 01 00 10 0d 0b 0d r  e  m  o  t  e  2e m  e  t  h  o  d  00 02 06 2e
<- 01 00 00 00 2d   

MAP_DEFAULT: remote.interface 
-> 02 00 0c 0b 23 09 i  n  t  e  r  f  a  c  e 
<- 02 00 00 00 2b 0c 23 09 i  n  t  e  r  f  a  c  e  01 03 00 07 0f 01 10 0d 01 0d 28   

MAP: remote_interface remote.interface (relation) 
-> 01 00 13 0d 0b 10 r  e  m  o  t  e  2e i  n  t  e  r  f  a  c  e  00 02 06 2b
<- 01 00 00 00 2a   

MAP: test (interface) 
-> 01 00 09 0c 00 04 t  e  s  t  01 03 00 02 2b 00
<- 01 00 00 00 29   

MAP: int32 
-> 01 00 0a 0c 00 05 i  n  t  33 32 01 03 00 09 13 20 20 04 16 20 17 18 19
<- 01 00 00 00 33   

MAP: test.doSomething (method definition relation) 
-> 01 00 0e 0d 29 0b d  o  S  o  m  e  t  h  i  n  g  00 20 2e 0b d  o  S  o  m  e  t  h  i  n  g  01 00 33 05 p  a  r  a  m  01 00 33 03 r  e  t  01 00 33
<- 01 00 00 00 32  

All the above is purely related to the data type agreement required before actually making the real method call. Obviously this is expensive and would not be performed every time. The client would cache the mappings already performed so that they can be used later.

The most important aspect of the list of data types resolved above is that only the data types required to make the doSomething method call are resolved. Additional types may need to be resolved before calling another method. This allows the client to only resolve the parts of the interface it uses. There are numerous advantages to this which I'll cover in another post.

Finally the actual method call is made.

MSG:
-> 07 27 00 02 00 32 01 00 33 00 00 00 0a
<- 07 00 01 00 33 00 00 00 1e  

The final message request can be broken down as the following. It is defined by the remote.rpc.request type:

 07 – Message Request
 27 – Location identifier type 27 (index)
 00 02 – Location index 2.  This is currently uint16.  Could be modified to uvint28 to reduce a byte.
 00 32 – Method identifier.  Could also be changed to use uvint28.
 01 – Number of parameters.
 00 33 – First parameter is a int32.
 00 00 00 0a – The input value 10.

The result is a defined by remote.rpc.response and is broken down as follows:

 07 – Message response
 00 – In Error flag.  Boolean value.  False.
 01 – One parameter returned.
 00 33 – First parameter is int32.
 00 00 00 1e – The return value 30.

The remote.rpc protocol is very simple and is missing some elements that would be required for a UDP implementation. For instance an identifier so that requests could be matched up with correct responses would be required.

It should also be noted that remote.interface and remote.method define a style of RPC mechanism. Different meta data could be produced to define a REST like interfaces and protocols.

There's still plenty to be done to get the Internet Draft ready by the 19th of October. Hopefully the above provided a small insight to how Argot works and can be implemented on even the smallest of devices.

IETF and Squeezing the Meta Dictionary

In the last few month I've struggled to find the right direction to take Argot. I've looked at reviving the Personal Browser concept, investigated SCTP and a few other things. These are all good research areas for Argot, however, they take the focus away from the core Argot idea. I've now returned to the core of Argot with a renewed focus driven by the 6lowapp IETF working group.

The 6lowapp IETF working group is being formed to develop the application protocols that will form the basis for the “Internet of Things”. Argot was originally created with small embedded systems in mind; in fact, in October 2005 I blogged about reducing an Argot RPC server to 7kb. While Argot can solve problems in other domains, the “Internet of Things” is the best fit for the problems it does solve.

The current plan is to develop an IETF Internet Draft (I-D) which provides the rationale for Argot, the technical problems it solves and provide a specification. In addition, I plan on developing an example service using Contiki. A lot of work to be done before the 19^th October. Of course, there's no guarantee that Argot will become an RFC, however, I should at the very least receive some good feedback and allow Argot to fit into the application stack developed.

An important part of developing for embedded systems is size, so I've been squeezing the Argot meta data and developing ways to allow the Argot protocol to work on the smallest of devices. In doing so, I've also been improving the meta dictionary and removing a few niggling constraints.

The first change to help size was the introduction of the uvint28 data type. This is an unsigned variable length integer with up to 28 bits of integer data. It uses the high bit of each octet as a continuation bit. The integer can be between 1 and 4 octets. The 28 comes from the fact that the normal 32 bit integer loses 4 bits of precision to the continuation bits. This type has replaced the uint16 (unsigned 16 bit integer) in the meta dictionary and in doing so removes many zero bytes. It also removes the limitation of the 16 bit integer. The meta dictionary when encoded after this change is 985 bytes long and includes 29 data types.

The next change was to introduce a meta.cluster type. This is a group definition and allows each name to refer to a cluster. This allows meta.name to use cluster references instead of recording the full class names for each definition. This change introduces a few new types, however, overall it removes a lot of duplicate information. The result is that the meta dictionary is now 888 bytes long and includes 35 data types. I'm not superstitious, however it's pretty cool that the end result is 888 bytes long; a very lucky number in Chinese.

[Edit: After doing some testing I discovered a one off bug which was causing an extra byte of data in strings. The meta dictionary is now 859 bytes long. Not as cool as the 888 byte length, but it is even shorter which is great!]

The end result of these changes should allow a full service description to use from 3kb of data and 3kb of code. In the coming weeks this will be confirmed and tested using the same mechanisms developed back in 2005. That is, using a Java client with full Argot protocol stack and a cut down purpose built embedded Argot stack.

Depending on the application and size of device, 3kb of data may still be too large. To resolve this, I've been looking at the Argot type resolution protocol. Instead of storing the Argot meta data on the device, the device can simply report an URL or other host that contains the Argot meta data. This should in effect allow the device to report the full message definition of its services in less than 1kb of code and data.

For those interested, here's the new meta dictionary data definitions.

(library.list [

/* BASE_ID 1 */

(library.entry
 (library.base)
 (meta.cluster))

/* UINT8_ID 2 */

(library.entry
 (library.definition meta.name:"uint8" meta.version:"1.3")
 (meta.atom uvint28:8 uvint28:8
  [ (meta.attribute.size uvint28:8)
      (meta.attribute.integer)
      (meta.attribute.unsigned)
      (meta.attribute.bigendian) ] ))

/* UVINT28_ID 3 */

(library.entry
 (library.definition meta.name:"uvint28" meta.version:"1.3")
 (meta.atom uvint28:8 uvint28:32
  [ (meta.attribute.size uvint28:28)
      (meta.attribute.integer)
      (meta.attribute.unsigned)
      (meta.attribute.bigendian) ] ))

/* META_GROUP_ID 4 */

(library.entry 
 (library.name meta.name:"meta" )
 (meta.cluster))


/* META_ID_ID 5 */
 
(library.entry 
 (library.definition meta.name:"meta.id" meta.version:"1.3")
 (meta.reference #uvint28))


/* META_CLUSTER_ID 6 */

(library.entry
 (library.definition meta.name:"meta.cluster" meta.version:"1.3")
 (meta.sequence []))

/* META_ABSTRACT_MAP_ID 7 */
   
(library.entry 
 (library.definition meta.name:"meta.abstract_map" meta.version:"1.3") 
 (meta.sequence [
  (meta.tag u8utf8:"id" (meta.reference #meta.id))
 ]))

/* META_ABSTRACT_ID 8 */
 
(library.entry
 (library.definition meta.name:"meta.abstract" meta.version:"1.3")
 (meta.sequence [
  (meta.array 
   (meta.reference #uint8)
   (meta.reference #meta.abstract_map))]))


/* U8UTF8_ID 9 */

(library.entry
 (library.definition meta.name:"u8utf8" meta.version:"1.3") 
 (meta.encoding 
  (meta.array 
   (meta.reference #uint8) 
   (meta.reference #uint8))
  u8utf8:"UTF-8")) 


/* META_NAME_ID 10 */

(library.entry 
 (library.definition meta.name:"meta.name" meta.version:"1.3")
 (meta.sequence [
  (meta.tag u8utf8:"group" (meta.reference #meta.id))
  (meta.tag u8utf8:"name" (meta.reference #u8utf8))
 ]))
 

/* META_VERSION_ID 11 */

(library.entry 
 (library.definition meta.name:"meta.version" meta.version:"1.3")
 (meta.sequence [
  (meta.tag u8utf8:"major" (meta.reference #uint8))
  (meta.tag u8utf8:"minor" (meta.reference #uint8))
 ]))

/* META_DEFINITION_ID 12 */
 
(library.entry
 (library.definition meta.name:"meta.definition" meta.version:"1.3")
 (meta.abstract [
  (meta.abstract_map #meta.cluster)
  (meta.abstract_map #meta.atom)
  (meta.abstract_map #meta.abstract)
  (meta.abstract_map #meta.abstract_map)
  (meta.abstract_map #meta.expression)
 ]))


/* META_EXPRESSION_ID 13 */


(library.entry 
 (library.definition meta.name:"meta.expression" meta.version:"1.3")
 (meta.abstract [
  (meta.abstract_map #meta.reference)
  (meta.abstract_map #meta.tag)
  (meta.abstract_map #meta.sequence)
  (meta.abstract_map #meta.array)
  (meta.abstract_map #meta.envelope)
  (meta.abstract_map #meta.encoding)
 ]))

/* META_REFERENCE_ID 14 */

(library.entry 
 (library.definition meta.name:"meta.reference" meta.version:"1.3")
 (meta.sequence [(meta.reference #meta.id)]))

/* META_TAG_ID 15 */
 
(library.entry
 (library.definition meta.name:"meta.tag" meta.version:"1.3")
 (meta.sequence [
  (meta.tag u8utf8:"name" 
   (meta.reference #u8utf8))
  (meta.tag u8utf8:"data" 
   (meta.reference #meta.expression))]))

  
/* META_SEQUENCE_ID 16 */

(library.entry 
 (library.definition meta.name:"meta.sequence" meta.version:"1.3")
 (meta.array 
  (meta.reference #uint8)
  (meta.reference #meta.expression)))

  
/* META_ARRAY_ID 17 */
  
(library.entry
 (library.definition meta.name:"meta.array" meta.version:"1.3")
 (meta.sequence [
  (meta.tag u8utf8:"size" (meta.reference #meta.expression))
  (meta.tag u8utf8:"data" (meta.reference #meta.expression))]))
  
/* META_ENVELOPE_ID 18 */

(library.entry
 (library.definition meta.name:"meta.envelope" meta.version:"1.3")
 (meta.sequence [
  (meta.tag u8utf8:"size" (meta.reference #meta.expression))
  (meta.tag u8utf8:"type" (meta.reference #meta.expression)) ]))


/* META_ENCODING_ID 19 */

(library.entry 
 (library.definition meta.name:"meta.encoding" meta.version:"1.3")
 (meta.sequence [
  (meta.tag u8utf8:"data" (meta.reference #meta.expression))
  (meta.tag u8utf8:"encoding" (meta.reference #u8utf8))]))


/* META_ATOM_ID 20 */

(library.entry 
 (library.definition meta.name:"meta.atom" meta.version:"1.3") 
 (meta.sequence [
  (meta.tag u8utf8:"min_bit_length" (meta.reference #uvint28))
  (meta.tag u8utf8:"max_bit_length" (meta.reference #uvint28))
     (meta.tag u8utf8:"attributes" 
   (meta.array 
    (meta.reference #uint8)
    (meta.reference #meta.atom_attribute)))]))

/* META_ATOM_ATTRIBUTE_ID 21 */

(library.entry
 (library.definition meta.name:"meta.atom_attribute" meta.version:"1.3")
 (meta.abstract [
  (meta.abstract_map #meta.attribute.size)
  (meta.abstract_map #meta.attribute.integer)
  (meta.abstract_map #meta.attribute.unsigned)
  (meta.abstract_map #meta.attribute.bigendian)
 ]))

/* META_ATTRIBUTE_CLUSTER_ID 22 */

(library.entry 
 (library.name meta.name:"meta.attribute" )
 (meta.cluster))


/* META_ATTRIBUTE_SIZE_ID 23 */

(library.entry
 (library.definition meta.name:"meta.attribute.size" meta.version:"1.3")
 (meta.sequence [
  (meta.tag u8utf8:"size" (meta.reference #uvint28))
 ]))


/* META_ATTRIBUTE_INTEGER_ID 24 */

(library.entry
 (library.definition meta.name:"meta.attribute.integer" meta.version:"1.3")
 (meta.sequence []))


/* META_ATTRIBUTE_UNSIGNED_ID 25 */

(library.entry 
 (library.definition meta.name:"meta.attribute.unsigned" meta.version:"1.3") 
 (meta.sequence []))


/* META_ATTRIBUTE_BIGENDIAN_ID 26 */

(library.entry 
 (library.definition meta.name:"meta.attribute.bigendian" meta.version:"1.3")
 (meta.sequence[]))


/* DICTIONARY 27 */

(library.entry
    (library.name meta.name:"dictionary")
    (meta.cluster))
 
/* DICTIONARY_BASE 28 */

(library.entry
 (library.definition meta.name:"dictionary.base" meta.version:"1.3")
 (meta.sequence []))

/* DICTIONARY_NAME 29 */

(library.entry
 (library.definition meta.name:"dictionary.name" meta.version:"1.3")
 (meta.sequence [
  (meta.tag u8utf8:"name" (meta.reference #meta.name))
 ]))

/* DICTIONARY_DEFINITION 30 */

(library.entry
 (library.definition meta.name:"dictionary.definition" meta.version:"1.3")
 (meta.sequence [
  (meta.tag u8utf8:"id" (meta.reference #meta.id))
  (meta.tag u8utf8:"version" (meta.reference #meta.version))
 ]))

/* DICTIONARY_RELATION 31 */
(library.entry
 (library.definition meta.name:"dictionary.relation" meta.version:"1.3")
 (meta.sequence [
  (meta.tag u8utf8:"id" (meta.reference #meta.id))
 ]))

/* DICTIONARY_LOCATION 32 */

(library.entry
 (library.definition meta.name:"dictionary.location" meta.version:"1.3")
 (meta.abstract [
  (meta.abstract_map #dictionary.base)
  (meta.abstract_map #dictionary.name)
  (meta.abstract_map #dictionary.definition)
  (meta.abstract_map #dictionary.relation)
 ]))

/* DICTIONARY_DEFINITION_ENVELOPE_ID 33 */

(library.entry
 (library.definition meta.name:"dictionary.definition_envelope" meta.version:"1.3")
 (meta.envelope
   (meta.reference #uvint28)
   (meta.reference #meta.definition)))

/* DEFINITION_ENTRY_ID 34 */

(library.entry 
 (library.definition meta.name:"dictionary.entry" meta.version:"1.3")
 (meta.sequence [
  (meta.tag u8utf8:"id" (meta.reference #meta.id))
  (meta.tag u8utf8:"location" (meta.reference #dictionary.location))
  (meta.tag u8utf8:"definition" (meta.reference #dictionary.definition_envelope))]))

/* DICTIONARY_ENTRY_LIST_ID 35 */

(library.entry
 (library.definition meta.name:"dictionary.entry_list" meta.version:"1.3")
 (meta.array 
  (meta.reference #uvint28)
  (meta.reference #dictionary.entry )))

])

And just for fun, here's the meta dictionary encoded. The encoding is a mixture of hex and ascii. Ascii is only used for a-z characters.

23 01 1c 01 06 02 1e 01 05 u  i  n  t  8  01 03 09 14 08 08 04 17 08 18 19 1a 03 1e 01 07 u
v  i  n  t  2  8  01 03 09 14 08 1c 04 17 08 18 19 1a 04 1d 01 04 m  e  t  a  01 06 05 1e 04
03 2e i  d  01 03 02 0e 03 06 1e 04 08 2e c  l  u  s  t  e  r  01 03 02 10 00 07 1e 04 0d 2e
a  b  s  t  r  a  c  t  _  m  a  p  01 03 08 10 01 0f 02 i  d  0e 05 08 1e 04 09 2e a  b  s 
t  r  a  c  t  01 03 07 10 01 11 0e 02 0e 07 09 1e 01 06 u  8  u  t  f  8  01 03 0c 13 11 0e
02 0e 02 05 U  T  F  2d 8  0a 1e 04 05 2e n  a  m  e  01 03 13 10 02 0f 05 g  r  o  u  p  0e
05 0f 04 n  a  m  e  0e 09 0b 1e 04 08 2e v  e  r  s  i  o  n  01 03 14 10 02 0f 05 m  a  j 
o  r  0e 02 0f 05 m  i  n  o  r  0e 02 0c 1e 04 0b 2e d  e  f  i  n  i  t  i  o  n  01 03 07
08 05 06 14 08 07 0d 0d 1e 04 0b 2e e  x  p  r  e  s  s  i  o  n  01 03 08 08 06 0e 0f 10 11
12 13 0e 1e 04 0a 2e r  e  f  e  r  e  n  c  e  01 03 04 10 01 0e 05 0f 1e 04 04 2e t  a  g 
01 03
n  c  e  01 03 07 10 01 11 0e 02 0e 0d 11 1e 04 06 2e a  r  r  a  y  01 03 12 10 02 0f 04 s 
i  z  e  0e 0d 0f 04 t  y  p  e  0e 0d 12 1e 04 09 2e e  n  v  e  l  o  p  e  01 03 12 10 02
0f 04 s  i  z  e  0e 0d 0f 04 t  y  p  e  0e 0d 13 1e 04 09 2e e  n  c  o  d  i  n  g  01 03
16 10 02 0f 04 d  a  t  a  0e 0d 0f 08 e  n  c  o  d  i  n  g  0e 09 14 1e 04 05 2e a  t  o 
m  01 03 7  10 03 0f 0e m  i  n  _  b  i  t  _  l  e  n  g  t  h  0e 03 0f 0e m  a  x  _  b 
i  t  _  l  e  n  g  t  h  0e 03 0f 0a a  t  t  r  i  b  u  t  e  s  11 0e 02 0e 15 15 1e 04
0f 2e a  t  o  m  _  a  t  t  r  i  b  u  t  e  01 03 06 08 04 17 18 19 1a 16 1d 04 0a 2e a 
t  t  r  i  b  u  t  e  01 06 17 1e 16 05 2e s  i  z  e  01 03 0a 10 01 0f 04 s  i  z  e  0e
03 18 1e 16 08 2e i  n  t  e  g  e  r  01 03 02 10 00 19 1e 16 09 2e u  n  s  i  g  n  e  d 
01 03 02 10 00 1a 1e 16 0a 2e b  i  g  e  n  d  i  a  n  01 03 02 10 00 1b 1d 01 0a d  i  c 
t  i  o  n  a  r  y  01 06 1c 1e 1b 05 2e b  a  s  e  01 03 02 10 00 1d 1e 1b 05 2e n  a  m 
e  01 03 0a 10 01 0f 04 n  a  m  e  0e 0a 1e 1e 1b 0b 2e d  e  f  i  n  i  t  i  o  n  01 03
15 10 02 0f 04 n  a  m  e  0e 0a 0f 07 v  e  r  s  i  o  n  0e 0b 1f 1e 1b 09 2e r  e  l  a 
t  i  o  n  01 03 0f 10 02 0f 02 i  d  0e 05 0f 03 t  a  g  0e 09 20 1e 1b 09 2e l  o  c  a 
t  i  o  n  01 03 06 08 04 1c 1d 1e 1f 21 1e 1b 14 2e d  e  f  i  n  i  t  i  o  n  _  e  n 
v  e  l  o  p  e  01 03 05 12 0e 03 0e 0c 22 1e 1b 06 2e e  n  t  r  y  01 03 1e 10 03 0f 02
i  d  0e 03 0f 04 n  a  m  e  0e 20 0f 0a d  e  f  i  n  i  t  i  o  n  0e 21 23 1e 1b 0b 2e
e  n  t  r  y  _  l  i  s  t  01 03 07 10 01 11 0e 03 0e 22