Part 1. Requirements on a device description for OR.

In this part, I will try to formulate requirements of a general device
description, that will fit to Openremote's requirement for open and
universal standards. Most importantly, we should strive at a truly
universal solution, describing the devices and its commands (nothing
more, nothing less) in a universal way, usable not only by the present
project in its present form, but also for other applications, now
existing as well as in future.

First of all, the question of "canonical commands" (or "canonical
names for commands") needs to be addressed. Ideally there should be a
finite list of commands that the devices accept, with uniquely defined
semantics. For example, there should be no command named "power", it
is semantically not clear what "power" does (it is a noun, not a
verb); instead there should be the names "power_on", "power_off"
(these denote discrete power-commands) and the command "power_toggle",
which toggles the power status (on -> off, off -> off). Another
example: it must be clearly distinguished  between a "play" button
(discretely taking the player to a "play" state) and a
"play/pause"-button (toggling the player between play and pause
state). For a project involving a large number of both devices as well
as contributors, this is probably not completely realistic -- which is
not an argument against a partial solution! A sound compromise has to
be found, like having "standard" function having canonical names, and
for the rest, a contributor invents "free-form" names (or submit
suggestions for new "standard" names to a maintainer).

The list of ("canonical") command names must of course be in a machine
readable format. The syntax for allowed names has to be make
explicitly clear -- including, for example, what characters are
allowed. A command should have, in addition to its identifier,
(optionally) a longer prose description, as well as a short form for
labeling on a tight screen. Furthermore, there has to be a concept of
synonyms: "ok", "enter", and "select" should probably be considered
the same command, i.e., only on of those should be a canonical
command, the other being synonyms to the first one.

The format has to be in a format that is readable both for humans and
machines (the latter in order to generate commands directly from the
device descriptions). It may be "hosted" in some database (like MySQL
deployed in Beehive), but nevertheless import- and exportable in some
readable format. It should of course allow versioning
(including traceability etc.) and comments.

Since the objects that we want to control are the devices, the device
data base should contain devices, not remotes. (Strictly speaking,
LIRC often does it the other way around.) The possibilty to model a
particular remote (the very piece that you hold in your hand) is also
desirable, but should not be confused by modeling the device and the
commands it understands.

It is not necessary, nor desirable, to restrict the description to
commands that use IR light as their transmission medium. For example,
many RF-signals are, just as IR signals, nothing but pulse-with
modulated signals, with a radio signal carrier (common carrier
frequencies are 433, 828, and 315 MHz) replacing the IR light carrier
of ~950nm wavelength. Serial (and IP-) signals in general consist of a
short text string, possibly with some preable and/or postamble. They
may take arguments, and/or return values. Also web-api commands are
found in some devices, possibly taking arguments and returning
something. There are also a number of other protocols, X10, KNX,
Bluetooth,... that it would be desirable to support at some time or
another.

It is also quite possible for a particular device to accept the same
command (semantically) through several different media. (Like a DVD
player with both IR and serial; you can say "play" both through IR and serial.)

There should be a possibility to group commands
in more or less arbitrary groups, either on hard differences (e.g. the
media or different device addresses on IR signals), as well as more
soft differences.

In the interest of single-sourcing, there also has to be one (or more)
include or reference mechanism, for example if one remote can be used
to control several devices.

Consumer electronics often come in variants:
European/Americal/...-models, DVD and BD-Players come locked to
regions/zones, in some cases some hardware features are optional
(e.g. a serial interface). Also firmware versions can account for
relevant differences that should be taken into account in the device
description. For this reason, in order not to need a large amount of
almost identical files (which also would violating the single-sourcing
principle), it is necessary to have attributes with an if-semantic:
The command/command-set "frunobulate" is present only in the North
American version.

For commands that have a structure like IR-commands (protocol name,
device address, command number, possibly other parameter) this
structure should be transparent in the device description. It would
also be desirable to be able to mark a particular command number as
doing nothing: "In search of undocumented features, I have tried this,
and verified that it does not invoke anything." Finally, regardless of
the "smartness", it is desirable to be able also to store "dumb" raw
signals. (Some Canon cameras understand exactly two IR signals, each
consisting of exactly two pulse pairs -- it would be silly and
counter productive to find a smart representation for this.)

A "device" should contain one or more command sets, each having a type
of infrared, rf, serial, ip, web-api, x10, knx,... A command set consists of a
number of commands, having something in common (either hard as
device number, or soft as being useful...).

Finally: redundancy is -- as such -- not a bad thing, as long as it
does not violate single-sourcing, or introduces contradictions. That
is to say, having several different representations of an IR signal is
OK, and can be practical, as long as it is clear, e.g., that one
representation has been computed from the other, and not the other way
around. However, having several conflicting representation, where
no-one knows where the particular data comes from -- is of course
evil.

Part 2. An attempt at a pilot solution

I have been working on these issues, basically as a hobby, for some
years. My project, described here
(project homepage) contains attempted solutions for device
descriptions and infrared(/RF) protocols. The project is GPL3, while
protocol and files describing them (like DTDs) are in the public
domain.
To act as inspiration for OR, I describe my solution in some detail.
For the description formats, I have selected XML with a DTD. Of
course, using an XML scheme (as in the XML formats within OR) would
have some advantages, in particular to be able to describe
restrictions with more precision. The reason why I selected DTDs is
that I have used them for 10 years... Migrating to Schemes is
nevertheless desirable. (There are programs transforming a DTD (a
*.dtd file) into a Scheme (a *.xsd file). This is not the goal, but to
also use the additional possibilities in data modeling offered by the scheme.)
The "canonical" names are defined in the XML file commandnames.xml,
from which a Java enum command_t is created through an XSLT
style-sheet mk_command_t.xsl. This defines a Java enumeration,
containing the valid command names. Further rules for command names are
found as comments in that file. DTD here.

Device description format.

I have defined an XML format according to the requirements in the
previous part. Syntactically, the XML format is defined by the DTD. Although the DTD (through the element devices) allows several devices in one file,
there is typically one file per device. The W3C recommendation
xinclude is used, making it possible to reuse and "include" certain
elements in multiple places.  (Made possible just by declaring the
parser to be xinclude-aware.) There is also the possibility of
"tagging" elements and referring to them in other places (ids and
Idrefs in the XML terminology).
A device has one or more commandsets, each being of a particular type
(IR, RF, serial, IP,...). Each command is described in a natural way,
a serial/IP command has a transmit string, possibly with a preamble
and/or postable inherited from the command set it belongs to, an IR
command has a protocol name, a device number, a command number, and
possibly some other parameters. (These issues will be dealt with in more
detail in the next part.)
Also (boolean) attributes discussed in the requirement section are
present. (However, my presently software does not evaluate them :-\ )
Some commands take arguments. The device description can model
this. Also, in some cases, output is delivered. Presently, there are
no possibilities to state a particular parsing, or semantics, of the
returned values, just to state the number of expected lines in the
response. (For example, a "parse string" like the one used by the C function
scanf may be able to cover a large number of practically important cases.)

Examples

As an example, consider the Oppo DV-983 DVD player with serial
support. This is supported by Harc with the file oppo_dv983.xml. Its
commands can be downloaded directly from the manufacturer (hats off!),
both the infrared and the serial commands. As can be found in the
spreadsheet on the IR code, the device uses the
nec1 protocol, with device number equal to 73. This corresponds to the
first command set in the mentioned device file. The serial commands
form another commandset, subdivided into command groups, depending on
whether they take an argument and/or deliver output. Note that some
commands (for example play) are available both as IR and as serial
commands.
Other interesting examples are the *_dbox2.xml files (referring to the
German cable/satellite receiver known as dBox2 with the open source
tuxbox software), each containing two (sagem_dbox2.xml,
philips_dbox2.xml), or three (nokia_dbox2.xml)
different infrared command sets as well as an elaborate web-api command set.

Another very interesting example is the Denon
A/V-Receiver denon_avr3808.xml having several infrared command sets
using the denon protocol (which, ironically, is called the "Sharp
protocol" by the firm Denon), as well as several command sets using
the denon_k (Denon-Kaseikyo protocol). Then there is a large number of
"serial" commands, available through both the serial box as well as
through the (telnet) tcp port 23. With the use of XML referencing,
these commands are present in two different command sets, without the
need of repeating.

Suggestions for smaller projects/"Exercises for the reader"

The general structure of these device descriptions, plus of course the
fact that is is XML, makes it possible to generate a lot of
interesting results. Some of these are already implemented in the
current HARC software. This includes the possibility to, from a single
device description, generate Pronto CCF-Files (with actually operating
buttons) (using Tonto through its API), generating JP1 Remotemaster files
for JP1 remotes, extending the XML files with rendered signals in raw
CCF format ("XML export"). Also rem-files for IRTrans and LIRC
configuration files can be generated – in some cases using some
compact (non-raw) representation, but in most cases using raw signals.
Writing a cascading style sheet (css) for the XML files to be rendered
in an XML-capable browser should not take too long. The same goes for
a simple XML -> HTML translator (using e.g. XSLT), for documentation
purposes, or, extending with suitable hyperlinks, to be able to send
HTTP requests using the Controller API – just using XML
transformations. Generating simple controller - or iphone.xml-file – either
for testing or for inclusion in larger projects can also be done this way.

Part 3.  IR signals, representation and computing.

For this part, some basic understanding of IR protocols is useful, for
example as described in this article.

Abstractly, an infrared signal consists of number of on-off
periods of modulated (typically 35-40kHz) infrared light. The
information in the signal sits in the timing of the on/off periods. An
infrared protocol is a mapping of a few parameters (a device number,
possibly a subdevice number, a command number, and sometimes others)
to an infrared signal, in the sense above. The device number (and
subdevice) is (are) (typically) the same for all signals intended for
the very device, while the command number enumerates the different
commands/keys. Typically (but not always), one bits of the parameters
are mapped onto one on/off-pair, one for the bit being 0, another for
the bit being 1 (often called pulse-width-modulation, PWM). The bits
making up the parameters have to be transmitted in a well defined
order: the only practically used schemes are MSB (most-significant-bit
first), and LSB (least-significant-bit first). Examples of common and
well known protocols are the
NEC protocol (using PWM), the
RC5 protocol, using bi-phase modulation. A complete infrared signal
typically consists of a lead-in pair, the parameters coded in a
suitable way, and possibly a lead-out pair - typically there is a
period of "silence" between the "payloads", normally much longer than
the useful signal.

A "signal" in this form can, possibly after some "format-wrangling", be
sent to IR transmitting devices like the GlobalCache, IRTrans,
USB-UIRT, CommandIR, Iguana, ... (of course with appropriate drivers
assumed). It can not be sent to a LIRC server in its current
form. (I once submitted a patch, which was rejected since the
maintainer felt that the signals belonged in the configuration file,
and that my patch broke the concept.)
On the Internet, in particular in the
JP1-Forum, there is a vast
knowledge of IR protocols. In particular, John Fine developed the
so-called IRP notation (recently thoroughly documented by Graham Dixon
here)
This is a precise, human as well as machine readable, description of
the mathematical mapping taking some parameters to an IR signal, in
the sense above. This "notation" is not only useful for documentation
and human communication, but can also be machine parsed and interpreted to
automatically generate IR signals. For example, the previously mentioned
NEC1 protocol is described as:

Somewhat appalled by the syntax and (then, in the dark ages before
Graham's work) informal syntax and semantics I designed my own format,
correspondingly called xirp. It is an xml-based file format for
describing IR-Signals, considered as a mapping from a few numbers
(device-number, command-number/OBC/function-number, subdevice-number?,
toggle? etc) to an IR signal. It is quite "inspired" by irp, but does
not try to mimic it to 100%. (One reason being that, at that time,
Graham's specs did not exist.) I have implemented some stuff that
definitely can be used (actually, I use it daily) at least for the
cases and for the protocols I have chosen to implement. The focus has
not been on covering all possible present and future cases. However, I
am fairly confident that all realistic deficiencies can be quite
easily fixed by straightforward extensions. Java-classes both for
parsing and for rendering, for API-, command-line, and GUI-usage are
available. The xml-format presently uses a DTD as its definition;
however, a migration to scheme is on my to-do-list. (The reason is
simply that I am pretty fluent in DTD-construction, unfortunately not
yet on schemes.)

XML offers some advantages compared to the irp-format. You do not need
to write the parser yourself; you get a validating parser by using
some available package (for example included in Java >= 1.5). The
DTD/Schema defines the syntax, so there is (at least in principle) no
need to waste time writing manual documentation. (Of course, the
semantics still needs manual documentation; however, there are tools
available to help.) A number of well understood tools exist.
As an example, consider the xirp file for the NEC1 protocol.
Most definitely much longer! This is not the place to describe the
syntax and semantics in detail, however, I hope the semantics is
reasonably clear.

Summing up

Summing we, need some way to systematically describe and implement
different IR protocols, allowing extensions with reasonable
ease. Implementing these as a OO-hierarchy of Java classes might be
one possibly solution. (Actually, I first tried this some years ago -
but discarded the solution in favor of the above described
xirp-files.) A machine readable description can of course either be
interpreted, or "compiled" in the sense of generating code (in Java or
in another language) to be compiled. Of course, an agreed list of
protocol and their definitions is also needed. The names used on the
Internet are not registered, and sometime contradictory. Synchronizing
with the JP1-ers may be a good idea.

Remark

It can be remarked that the above applies to large classes of
RF-controlled equipment too (for example the
Intertechno radio controlled switches with
code wheel), since they are also effectively PWM-signals, just with a
RF carrier instead of an IR carrier, similarly using 35-40kHz
modulation.