All,
I am currently looking into the iPhone console and add some more features for KNX. To be able to have different kind of controls rendered on the iPhone screen, depending on the device which is controlled with it, we need a way to tell the iPhone what the control represents. This could be just an image for layout purposes, a button for a remote, a UISwitch for KNX or X10 light switch, a UISlider for a dimmer, ...
In the moment the schema supports the following xml:
<screen id="1" name="Number Pad" row="4" col="3" >
<buttons>
<button id="1" label="1" x="0" y="0" width="1" height="1" icon="1.png"/>
<button id="2" label="2" x="1" y="0" width="1" height="1" icon="2.png"/>
</buttons>
</screen>
I would propose to rename <buttons> into <controls> and <button> into <control>. The iPhone source already calls it a control anyway
This way the name is more general a can represent different UI elements and not just buttons.
To distinguish between the different type of controls we could add an attribute "type" which can be e.g. "button", "image", "knxSwitch", "knxDimmer", ...
The iphone.xml could look like this:
<screen id="5" name="Lights" row="5" col="4">
<controls>
<control id="40" type="button" label="Up" x="3" y="1" width="1" height="1" />
<control id="41" type="image" label="" x="3" y="2" width="1" height="1" icon="Lightbulb.png"/>
<control id="42" type="button" label="Down" x="3" y="3" width="1" height="1" />
<control id="43" type="knxSwitch" label="Dining table" x="0" y="4" width="1" height="1" />
</controls>
</screen>
Also, for defining a background image on an activity, we need an "image" attribute for the activity.
Right now I have implemented the "image" object and the "knxSwitch" object. The "knxSwitch" is rendered as a UISwitch which is the On/Off control which you know from the preferences, for example. My "knxSwitch" is calling the REST servlet when it is created to retrieve the current state of the light. For this to work I created a new CommandType "STATUS" on the server side which queries the state of the configured switch from KNX.
We will also need a way to inform the iPhone console if a satus is changed from a switch in the wall so the UI can update it's state.
Any ideas?
--Marcus
Comments (19)
Sep 02, 2009
Juha Lindfors says:
Thanks Marcus, I've got some feedback to this, I'm writing it down to a more fo...Thanks Marcus,
I've got some feedback to this, I'm writing it down to a more formal document right now. I'm thinking of a bit more extensive overhaul of the XML schema, to set the roadmap even if its not all implemented in one stage. I'll post initial draft here and then move it to a separate wiki page after feedback and eventually to implementation milestones.
Sep 02, 2009
Marcus Redeker says:
Sounds great. Let me know if you need any more input. --MarcusSounds great. Let me know if you need any more input.
--Marcus
Dec 01, 2009
Juha Lindfors says:
Below is a proposal for a new XML schema for panel UIs. It is posted here first ...Below is a proposal for a new XML schema for panel UIs. It is posted here first to collect feedback and manage discussion. The goal is to turn it into a full wiki page with the spec and create a proper XML schema to back it up. Then proceed to defining implementation milestones.
The document attempts to collect a number of requests and recent discussions. It is important to note that this document is intended as a longer term roadmap for the panel UI definition. It is expected to be implemented over several milestones and versions, not all at once. While the XML schema definition may be fairly complete at this time, the actual panel implementations may implement it partially. It is intended as a guideline to features to come.
There are several areas in this schema proposal that are still incomplete. Welcoming proposals (as complete as possible) to fill those gaps. The highest priority items are more precisely defined at the moment. If you want to bump up your favorite UI feature closer to implementation and inclusion in the trunk, start first by defining the feature here in the XML schema.
Feature implementation priorities have been marked where possible. There are several to-do tasks for lower priority items. If any of them is of interest to you, you can bump the feature closer to trunk by providing an implementation. However, make sure your feature first has an accepted definition in the XML schema as pointed above.
Thanks to Marcus Redeker and Andy Oliver for already providing initial feedback.
Schema Elements for Consideration
Considering the following UI elements in this version of UI XML schema:
On the layout side, absolute positioning of components vs. grid should be separated as their own layout components:
Layout element(s) nest the visual UI elements. The visual UI control implementation for the proposed XML tags is basically the one and the same UI control if implemented from scratch. However, a platforms native components may be used instead if they fit into the spec:
In terms of priority, changes to <screen> and implementations of <navigate>, <monitor>, <grid> and <image> are higher, then follow with additional features to <button> and finally <toggle> and <slider> implementations.
Outside of Consideration
A couple of items this document does not yet address:
[Edit: add link to gesture proposal]
[Edit: add animation to the list of out-of-scope features]
Monitor Polling
The "active" (stateful) UI components <monitor>, <toggle>, and <slider> can be updated by the controller (via events triggered by devices or the controller itself). While pushing of state changes would be most efficient, we are opting for a HTTP based request-response protocol and not assuming the client applications are capable of serving incoming connections. Therefore a HTTP polling should always be available for console applications. Alternative protocol implementations can be addressed at later stage.
Given the inefficiency of polling in general, individual components should not generate HTTP requests to the controller but an aggregate HTTP request should be sent from the panel on a given interval to retrieve updates for all relevant UI components. In order for the controller to track which events are relevant to any given panel UI, it must have a list of relevant UI component IDs. This could be implemented in two different ways: registering relevant UI IDs with the controller creating a stateful session between the panel client and controller, or passing the relevant UI IDs per HTTP request in a stateless protocol implementation. At this stage of development neither implementation strategy has obvious implications as high availability of the controller has not been addressed yet. Both implementations are feasible although a stateless protocol would seem to have a slighly smaller implementation cost, albeit at a cost of higher bandwidth use.
Controller API Changes
Controller REST API must be updated in sync with the panel software changes where necessary. For active component implementations an additional API must be developed for HTTP polling. The details of that are left for another thread/document.
Designer Tool Changes
UI Composer can be developed independently and after the new panel UI components are completed. New panel UI's can be tested and deployed by technical users by modifying the relevant configuration files prior to UI composer changes.
Changes to <screen> Element
Screen is currently implemented as defined below:
A new attribute "background" should be added to define a background image for a screen.
<xsd:attribute name="background" type="xsd:string" use="optional"/>When nothing else is specified (see "Other Attributges"), the background image should be drawn from the top-left corner in its original size. The z-order for a screen background image should always be lower than any other UI component on the screen.
Replacing Row and Col Attributes with <grid>
The existing "row" and "col" attributes in <screen> definition should be replaced by a nested <grid> element.
Other Attributes
Additional optional attributes for screen background are "bgcolor" for setting a screen background color separately from the default black, "background-repeat" and "background-position" (cf. CSS spec) for additional control of how background images can be placed. These attributes are not high priority items at the moment but would accept contributions if there's more interest in getting these features into a release.
Screen Grouping
Screens should be enabled for arbitrary grouping which can be used in navigating the UI. A group has a name and screens can be ordered within a group. One screen can belong to multiple groups – e.g. a screen for controlling lights in the kitchen could be grouped as "Lights", "Kitchen", "First Floor", "Building 13". The <group> elements essentially replaces the existing <activities> element with a more generic grouping for screens. Group names can also be used in navigation UI elements within <navigate> element.
Changes to <button> Element
This is the only UI component currently implemented. It should remain as-is with some additional new features.
The current definition is below:
The "label" attribute can stay as it is. It is used if no icon image is defined.
Coordinate attributes "x", "y", "width", "height" should be regarded as grid positions if the <button> element is nested within a <grid> tag. Otherwise they should be considered as absolute screen positions. Attributes "width" and "height" should be optional. If they're not defined then within a <grid> one cell size is assumed. When absolute positioning is used the width and height is whatever the implementation considers as "appropriate". If "icon" images are used and either "width" of "height" is omitted then the actual image width and height should be used. If "width" and/or "height" is specified, only then should the image be scaled to that size.
UI elements specified with <button> should not visually overlap when used within a <grid> element and do not specify a z-order. The only exception is when an icon image is used and no "width" or "height" attribute is specified (as original image sizes should be preserved), or when absolute screen position is used. In such case visual overlapping may occur but the z-order and overlapping behavior should be unspecified.
The bounding-box (area which is active to user interaction) of a button is rectangle-shaped. A button's bounding box should be active and react to user interaction regardless whether there's a visual presentation drawn or if image opacity is used. Other pre-defined bounding-box shapes could be added with relatively low effort, if necessary. Free-form bounding area is left for later iterations.
New Attributes for <button>
Attribute "repeat" specifies if the button implementation should repeat the command while the button is pressed or measure the length of time a button is pressed instead of reacting only to the button "release" event. Use of HTTP request-response protocol makes the implementation here a bit challenging. An initial implementation already exists in the controller API to support this operation.
<xsd:attribute name="repeat" type="xsd:boolean" use="optional"/>Attribute "onPressIcon" allows a secondary icon image to be specified that should be drawn when the button is pressed. This gives a visual feedback when UI event is received.
<xsd:attribute name="onPressIcon" type="xsd:string" use="optional"/>Actual panel implementation is not a high priority but should be relatively easy to add as a contribution if there's more interest in this feature.
Other Features
Other currently low priority but would consider as contributions items:
Allow <label> element to be nested within a button. This replaces the "label" and "icon" attributes in <button> with a label that can be stylized with regards to font, line-, fill- and background colors.
Similarly, would consider elements to stylize the default button face (when images are not used) with fore- and background color and border-style.
New <image> Element
Allows an image to be placed on the screen layout. Either in an absolute position on screen or within a <grid> element.
The "x" and "y" attributes should be interpreted either as absolute screen position or a grid cell position if <image> appears nested within a <grid> element. If "width" or "height" is omitted, source image original dimensions should be used. In case the original image size leads to overlapping of elements, the behavior is unspecified. The <image> element z-order is higher than the image specified as <screen> background image and lower than UI elements such as <button>.
The <image> element does not map to any events and does not react to user interaction.
New <navigate> Element
The <navigate> element behaves similarly to <button> except it doesn't generate a HTTP request to the controller but instead updates the UI to a new view. Proposed definition below:
Coordinate attributes x,y, width, height and attribute "icon" behave same as with <button> element. Attribute "toGroup" allows specifying a screen group name the UI view should update to. Attribute "inGroup" allows a logical name for the screen (set of "next", "previous", "first", "last"), screen name or index number to be specified within a group. If no "toGroup" has been specified, the same group where the <navigate> element exists in is assumed.
The "hint" attribute allows a system-specific string to give a hint to the panel platform if the navigation element should integrate with the platforms native UI navigation elements (menu, toolbar, etc.)
New <grid> Element
The layout element <grid> is used to specify a table-style layout where each UI component is constrained to a cell in a grid.
The <grid> element coordinate attributes "x" and "y" are the top left corner position of the grid on the screen. The values may be absolute locations on the screen or relative location expressed as a percentage value from the top-left corner of the screen. Attributes "width" and "height" define the size of the grid also either as absolute size on screen or relative percentage values. Attributes "rows" and "cols" define the number of cells in the grid. Each cell is equally sized.
The attributes for the nested <cell> element are x and y cell positions within the grid. A cell may span multiple units of the grid horizontally or vertically by defining a width or height above value 1. If nothing is defined, a single cell unit size is assumed.
New <monitor> Element
This is a to-do item.
New <label> Element
This is a to-do item.
New <toggle> Element
This is a to-do item.
New <slider> Element
This is a to-do item.
Implementation Priorities
Other Considerations
Would welcome feedback on if and how well the proposed XML schema maps to native Android XML UI bindings and to GWT UIBinder bindings.
Sep 08, 2009
Marc Fleury says:
So if someone doesn't define a grid the positions for buttons are absolute. It c...So if someone doesn't define a grid the positions for buttons are absolute. It confused me for a while that the same x,y could represent 2 different things depending on whether we had defined a grid or not. Line 77 in the xsd ("The meature is one cell on the iPhone panel") was not clear.
Sep 08, 2009
Juha Lindfors says:
Yes, that is correct. We could have explicit tags for all layouts including <...Yes, that is correct. We could have explicit tags for all layouts including <absolute>, if that makes it clearer. It does bring more consistency to the schema.
Yes the XSD is not sufficiently documented in the 1.0 version. This thread and document should make the 2.0 version of the schema better in that regards.
Sep 08, 2009
Marc Fleury says:
Sure thing, this is a big step forward, I particularly like how you have handle ...Sure thing, this is a big step forward, I particularly like how you have handle the navigation. Basically any application on a device is a tree of navigation with leaves that are devices. This way we can handle things such as your ideas about room->remote->device (a la bobby) or some of the navigation that Marcus is talking about. Of course a clean XML is the central part, a rich UI to assemble them is the other. I was thinking that the current navigation does in fact get you to the goodies, including future demos, in few clicks which is good.
When you talk about Layouts, saying we won't do anything but <grid> and <absolute>, I assume you refer to things such as the java gridback layout? fancy layouts like that are overkill for HA application IMHO. I would be curious to see needs for layouts beyond the 2 we would offer.
Sep 08, 2009
Juha Lindfors says:
When you talk about Layouts, saying we won't do anything but <grid> and &...Yes, correct again. Android currently defines some others (Common Layout Objects) but I am waiting on a clear use case that shows how these others are needed before spending the effort in defining or implementing them. So beyond <grid> and <absolute> has no priority right now but would consider quality contributions, of course.
Sep 11, 2009
David Reines says:
In regards to the GWT UiBinder service, a widget defined via the UiBinder consi...In regards to the GWT UiBinder service, a widget defined via the UiBinder consists of 2 artifacts. A UI xml file that defines the widgets structure and a Java class that binds the data to the widget. So, a simple tranformation of the iphone.xml to a GWT UI xml is not sufficient. The process will have to generate some Java code as well.
Another thing to keep in mind, is that the UI xml files are interpreted at compile time, not at runtime. So, if we do decide to use the UiBinder service, then a process will have to invoke the GWT compiler to generate the JavaScript defined via the UI composer. Then, at runtime, the web console app would utilize the user's profile information to serve-up the appropriate JavaScript. On the positive side, since there's no downloading and parsing of the iphone.xml, this solution provides a much snappier initial load of the console app. Which is a really nice benefit.
Given the performance upside of this solution, I'm going to spend some more time looking into the UiBinder service. I'm planning on building a more sophisticated proof-of-concept utilizing UiBinder. Hopefully, I can find a way to simplify the requirements of the generation process. For example, maybe there's a way to generate a single widget that simply assembles a set of re-usable widgets.
Any thoughts or feedback would be appreciated. It's always great to get others perspectives on a problem.
-Dave
Sep 14, 2009
Juha Lindfors says:
Thanks David, Interesting findings. Thanks for looking into this. Let us know i...Thanks David,
Interesting findings. Thanks for looking into this. Let us know if you make further progress on this.
– Juha
Sep 20, 2009
Juha Lindfors says:
An issue with this proposal is that it still does not properly separate user int...An issue with this proposal is that it still does not properly separate user interface elements from actions (or view from control in MVC parlance).
It should be modified to allow UI views to be constructed as aggregate controls, for example:
In the above example, roles are clearly separate where <button> is purely an UI view element and <command> and <navigate> are control elements (note that command id may map to a macro sequence on the controller). In case of an user interaction through a button, a command may be executed and the view updated to a different screen.
– Juha
Sep 24, 2009
Juha Lindfors says:
Below is a sample XML instance based on the schema proposal (with some feedback ...Below is a sample XML instance based on the schema proposal (with some feedback and changes baked in). This sample doesn't contain all the new features discussed but demonstrates the basic elements, replicating the current button-based UI model.
Sep 24, 2009
Marcus Redeker says:
How will this be rendered? Do we have three "old activities" which are represent...How will this be rendered? Do we have three "old activities" which are represented by the groups?
Which is the "start screen" in this example?
If we would not have the screens inside a group would each not grouped screen represent an "old activity"?
Where would the navigation elements be placed if we want the three groups to be shown inside a TabBar?
Is the zIndex used in case buttons are overlapping?
Sep 24, 2009
Juha Lindfors says:
Do we have three "old activities" which are represented by the groups? No. If ...No. If you want to have an activities screen then its something that should be explicitly defined in the UI definition as navigation elements. I'm trying to move away from hard-coded UI assumptions (initial activities screen being one of them).
If you're keen on building automated activities-like screens then that should be implemented as a functionality in the UI designer which generates the appropriate XML definition for "activities" based on group definitions and is then rendered by the panels. This is more of a templating functionality in the designer then.
This is following the same principle as with moving gestures as explicit declarations in the UI definition.
First screen of the first group as an implicit order. We could add an attribute for explicit start screen although I'm not sure it is necessary. Will consider adding it to the schema anyway.
No. We should mandate a minimum of one group definition.
Still working on that. Will update later. Within the <groups> section looks like a good probability.
Yes, although having buttons overlapping themselves (rather than overlapping images) is a bit questionable use case. It can be a bit of a challenge for UI rendering frameworks as well. Still thinking whether an explicit z-index is even necessary.
Thinking about moving the image attributes from button as standalone <image> elements inside <button> though.
Sep 24, 2009
Jean-Luc Vanhulst says:
I think navigate should have the option to go to a screen as well. Across a grou...I think navigate should have the option to go to a screen as well. Across a group.
And an extra aggregate level is needed
<Group> (Room)
<Device> (Or call that 'Activity"?)
<Screen>
Many devices will have more than screen (like a TV) and they must be grouped
Sep 24, 2009
Juha Lindfors says:
I think navigate should have the option to go to a screen as well. Across a gro...It does. It accepts both toGroup and toScreen attributes.
Screens can belong to multiple groups (building, floor, room, device, all my tvs, all my A/V, etc.)
Sep 24, 2009
Jean-Luc Vanhulst says:
Okay, I'm thinking about (eventually) saving and restoring 'pre made' device (li...Okay, I'm thinking about (eventually) saving and restoring 'pre made' device (like the Bobby / cremote).
If I create a nice device with a few screens I think it makes sense to have them as a 'group', but it's actually a 'device'.
For example a decently designed complete TV remote will be 2 or 3 screens. They belong together.
And that device with 3 screens is part of a few devices in my Group "Living room".
I can see some use for a screen across groups but in general it wouldn't work like that because I have a TIVO in my bedroom and in my living room I will need different commands attached to each button, UNLESS we create exactly the DEVICE concept where the screens belong to this thing called TIVO that comes with commands and is physically connected to a device.
There's a long discussion somewhere else on that.
Sep 25, 2009
Yu-Chung Wang says:
Do we want to add an extra title for each elements? In this way, we don't need t...Do we want to add an extra title for each elements? In this way, we don't need to design a lot of icons with text. Icons is good. However, it is not intuitive someytimes for users to know its meaning. In addition, too many icons will slow down the loading process as well if the number of icons in all screens goes high.
Sep 25, 2009
Jean-Luc Vanhulst says:
What do you mean with extra title? Icon is optional, without that "Label" is use...What do you mean with extra title? Icon is optional, without that "Label" is used as display name? So use "Button" without an image and fill in "Label"
Sep 25, 2009
Yu-Chung Wang says:
Sorry. I don't notice the label field before. It is exactly what I need.Sorry. I don't notice the label field before. It is exactly what I need.