Embedding Natural User Interfaces

Have you ever been given a video file from a visual designer and asked to recreate the complex scene as an animated UI component? Then, after weeks of frustration between you and the designer, everyone compromises to implement a designed deemed ‘close enough’. If you are developing on a modern platform such as iOS, Android, BB10 or Windows 8 this has happened to you. Welcome to the new world of developing NUIs. But it doesn't have to be this way.



A Natural User Interface (NUI) is a fluid, dynamic, motion-based UI that aims to make technology effectively invisible. It is based on natural human behaviours and reactions. Individuals quickly become experts at using an NUI since the focus is on ease of use, not development. NUI is an integral part of any modern user experience: natural feeling, intuitive, responsive and easily integrated without a large learning curve.



Developing An NUI



As a developer of NUI applications, you’ll be working more with designers and user experience experts. They’ll push the boundaries of what’s possible with current UI frameworks. The designer’s vision is becoming more important than the developer’s restrictions, because the user experience represents the window to all modern products. They serve not only as differentiation within a competitive landscape, but also a way of gaining love and loyalty for a product. The trick is the ability to recreate this original, unique and beautiful vision of the designer in code.



Most current tools require that each component is coded, mimicking the look and feel desired by the designer. Each animation with every little inflection and bounce has to be recreated, which takes a lot of time. And you probably won'’t get it right the first time.



It is likely that even after many hours or even days of work, you’ll end up with something that still doesn’t appear as the designer envisioned. You’ve spent extra time working on the look of the icon, object or larger application, taking precious time away from implementing features or ensuring software stability. YOUi Labs solves this problem with our uSwish framework and unique application development workflow.



Design Workflow



YOUi Labs created a plugin for Adobe After Effects, the industry-leading design software package. With this plugin, you can import graphics and animations into your uSwish application exactly as they are created by the designer. There’s no need to fuss with images and equations. With almost no code, you’ll be able to load up the designer’s graphics and everything will look right and as the designer had really intended.



Adobe After Effects can be used in 2D or 3D to animate and compose media. It is often used by designers to provide video files showing the desired look and feel of their design vision. In the uSwish workflow, the designer exports these files using the uSwish plug-in without the need to learn any other design tool.



uSwish aims to support all applicable Adobe After Effects features, currently supporting solids, null objects, parenting relationships, all layer transformation properties, masking, clipping, lighting, shadows, and keyframes with interpolation types.



One tool commonly used by designers but not supported by Adobe After Effects is nine patch images. The uSwish framework supports an advanced version of nine patch images called N-Patch. Nine patch is a standard way of representing images that can be scaled while retaining important information, like edges or other details that should not be stretched. While traditional nine patch images support only nine patches, the uSwish N-Patch can have an arbitrary number of patches. We provide a special Adobe After Effects filter to visualise how your N-Patch image will look when loaded by uSwish, which can be used by the designer to visualise how the image will appear in the uSwish engine.



A scene view in uSwish defines the logic of a component. Scene views are defined as C++ classes in uSwish, which corresponds to a composition. You will need to tag each composition in the file with the uSwish scene view to which it corresponds. These classes can be views provided with uSwish or custom derived views created by a developer.



Scene views define timeline labels corresponding to labels in the timeline panel. The scene view uses these labels to play the correct animation under certain conditions. These animations can look however you would like.



uSwish uses a scene tree to organise the nodes in the UI. This corresponds to the parenting relationship set in Adobe After Effects. Transformations are carried down the scene tree mimicking how they are applied so what you see there will be reproduced exactly in your uSwish application.



For example, a push button defines a component that can be clicked. The most basic implementation defines labels for a down animation and an up animation. The internal logic of the push button plays the “down animation” when the user presses down, and the “up animation” when the user releases the button. If one or both animations are missing, there will simply be no visual effect on the corresponding action.



Building Application Logic



Scene views typically provide ways of interacting with other components in the UI. They also define functions that allow other controls or the main application to manipulate them. For instance, when a push button is clicked, it emits a signal, allowing the application to respond to button clicks. A push button also provides enable and disable functions allowing external components to manipulate the availability of the button to be clicked.



Using the Observer pattern, you can define your application logic to react to scene view outputs, or have scene views interact with each other. The uSwish library includes a signal and slot mechanism that is used by many pre-built scene views, allowing you to easily connect the outputs of views to other views, or even functions you create.





The most basic component is a button with an 'up' and 'down' animation



Another way to define the logic of your application is to create custom scene views. The benefit of this approach is to separate component logic from application logic. Custom scene views are reusable, and they can be used like any view provided by uSwish. A custom scene view is derived from CYISceneView or a subclass. It can define custom animation labels, input functions or output signals, defining new ways for your components to interact with each other within the application.



Dynamic Content And Layout



If you’re familiar with Adobe After Effects you may be wondering how it can be used to lay out dynamic content. Positioning of graphics in Adobe After Effects is absolute and pre-defined, while NUI applications are typically more dynamic.



Designers can work with placeholder content in Adobe After Effects that provides the look of dynamic views. The parts that will be dynamically populated are typically given meaningful names in the composition. These nodes can then be looked up in code by name and the placeholder images and text can be replaced with real content.



Adobe After Effects does not have to be used to lay out the entire application. The designer can create individual components — each as a composition that can be tagged with scene view class names before exporting. These can then be instantiated in code as necessary.



Due to the hierarchy of nodes, animations can be defined relative to the node rather than its parent. This means that the animation will not have absolute positioning and therefore support a dynamic layout.



In our sample application, you can see examples of both of these concepts. The Adobe After Effects file defines what a heading looks like and what a content row looks like, both with placeholder content. The application initialisation sequence creates as many headings and content rows as necessary and fills in each one with real content.



The parent — the scrolling list — takes care of positioning these items, and the animations on each item occur relative to the item itself.



Platform Support



Whether you develop your application with one platform in mind or many, it is easy to build a uSwish application and have it running on multiple platforms. uSwish is developed in C++, which is supported by many platforms either directly or by hooking into another framework. We expose bindings to other languages, for example Android, so that UI development can be done in the default platform language.



YOUi Labs designed and wrote uSwish to support development for Windows, Linux and Mac OS X. While Eclipse is typically used to develop Android apps, and Xcode to develop iOS apps, you are free to set up your environment however you choose. Eclipse, Visual Studio and Xcode are directly used by the development team at YOUi Labs.



Adobe After Effects runs on Windows and Mac OS X, and the uSwish export tool is supported on both platforms. The exported files are platform agnostic.



Hardware Considerations



Touch-based interfaces heighten user sensitivity to interactions. Devices using such interactions must be able to deliver a responsive experience. The user’s input must be detected, processed, and the result displayed promptly, without any noticeable lag. These tasks are processed on the CPU and can become delayed when the processor is heavily loaded. This is particularly important for NUI applications that are fluid and movement-based.





Animations are able to support a dynamic layout



If a lot of pixel processing is taking place, as tends to be the case in an NUI application, some of this work can be offloaded to the GPU. The GPU is designed as a Single Instruction Multiple Data (SIMD) style processing array, meaning many processors working in parallel on the same simple task. This achieves high performance when a lot of data is available. An NUI involving complex visuals and plenty of movement can benefit from this.



RAM speed and the number of cores in the system’s GPU and CPU should be well matched. One high performing component may lead to a bottleneck in the other, thus improving one system specification may not increase the overall performance of the system. YOUi Labs understands these limitations and can help you make the most of your target hardware.



An application built with uSwish will look exactly like the designer’s vision with no need to code every detail. uSwish has been designed to achieve high performance as a cross-platform engine or tailored to a particular platform depending on your needs.



uSwish is a powerful yet simple to use engine aimed at creators of cutting-edge NUI applications. It allows for the implementation and execution of a designer’s vision at a fraction of the time and headaches of traditional workflows.



Author Profile: Kelsey’s success in UI Development is based on her special and sought-after blend of Fine Arts and Computer Science. Her passion for perfecting usability in technology shines through in all the projects she has undertaken. Earning a Bachelor’s of Math at the University of Waterloo, Canada, Kelsey is well positioned in applying theoretical concepts to real-world problems. After holding various positions in design, software QA, UI testing and development, Kelsey currently focuses on user experiences across Natural User Interface developments for the smartphone, tablet, ipTV and digital signage markets.