WOW! Can you believe it?! Adobe MAX Day 1 is complete, and we have seen a ton of new and exciting announcements. It's hard to think we have more to go. Being a die-hard Flash developer, you can guess which ones are my favorite: Flash to iPhone publishing, and Flash Player 10.1. A child of the Open Screen Project, Flash Player 10.1 brings a bunch of slick new features, especially for a dot release. Requests from the community landed us features such as HTTP video streaming, content protection, global error handling, and finally, multi-touch support. With the new iPhone publishing options in Flash CS5, you really can write once and deploy anywhere if you so choose! What's so great about the Open Screen initiative and Flash Player 10.1 is that while it has seen an unprecedented amount of support on the mobile platform, it's not just for mobile devices. As Adobe CTO Kevin Lynch demonstrated during the MAX keynote, touch and widget-enabled televisions, touchscreen tablets and PCs all get in on the new multi-touch hotness.

Now that we know we're going to have these sexy new toys at our disposal, we need to examine what exactly multi-touch is and some application design concepts that we should consider before implementing multi-touch in our Flash Player 10.1 projects in the future.

"There is no reason in this day and age, that we should be conforming to a physical device.[...] We have so much technology these days, that interfaces should start to conform to us." - Jeff Han.

That quote is from 3 and 1/2 years ago, during Jeff Han's unprecedented demo of some spectacular multi-touch interfaces at TED. Up until that point, many of us had only seen such technologies in movies like Minority Report. There have been multi-touch installations since the 80's, but we had no idea then that so much work was being done to get where we are today. 2006 was also the same year that the iPhone debuted, a product and platform that changed the way we think about how we interact with computing devices. Since then, we have seen advanced touchpads on laptops, touchscreen desktop PCs, an explosion of touch-based mobile devices, and even fully operational touch tables like Microsoft Surface. The one thing missing from the headlines of that history-changing tech revolution was the dynamic power of the Flash Platform. Not anymore.

The good news is that it's the Flash Platform's turn now. A new era of the Flash Platform is here, and bringing with it countless development opportunities in Flash Lite 4, AIR 2.0, and AIR for Mobile, all of which support multi-touch in some way. In fact, there were at least a handful of sessions at Adobe MAX 2009 dedicated to Flash Player and multi-touch, including one lead by Danny Dura and Matt Bugbee, which I highly recommend you watch. They even built a Flash-powered multi-touch display for the Adobe MAX 2009 Twitter Companion app.

A personal epiphany that I have had since I began research on this topic a few months ago, is the notion that "Flash was always meant to be multi-touch". I got into Flash development after extended experimentation in digital art, when my attention turned to User Experience, User Interface, and Interaction Design. Responsiveness, personalization, humanity, and enjoyment are hallmarks of a well-designed user experience, something that Flash has always been highly adept at producing. Now, with multi-touch, the limits of creativity have been pushed to a point where Flash can be used as tangibly and ubiquitously as it was meant to be.

It's my hope that Flash/Flex/AIR developers aren't underestimating just how much this will revolutionize how we work and how important testing hardware devices will become. Director of the Ribbit Developer Platform, Chuck Freedman, has an excellent post on how multi-touch in Flash Player will introduce the era of "2'face development," where we as Flash Platform developers need to be able to manage progressive enhancement and graceful degradation with devices that don't yet support multi-touch. So, we as developers need a good foundation of knowledge before we start buying hardware devices and writing applications, guns-blazing, all willy-nilly.

That said, we are on the edge of an entirely new "2'face" era of Flash development that is sure to be the most fascinating and opportunistic yet. The good news is that we won't have to wait much longer. Getting new native touch APIs in Flash Player is a gigantic leap in the number of environments and devices we can deploy our apps to. It's the best time ever to be a Flash developer.

Let's jump right in and take a look at some application design concepts and best practices, shall we? Raise your hand if you have heard of HCI, or Human-computer interaction. Nobody raised their hands. Well, let me explain. According to Wikipedia, HCI "is the study of interaction between people (users) and computers. It is often regarded as the intersection of computer science, behavioral sciences, design and several other fields of study. Interaction between users and computers occurs at the user interface (or simply interface), which includes both software and hardware..." Let's dissect that description. If you noticed, the description is very specific when mentioning related disciplines. 'Computer science' is obvious. The programming and cross-talk between software and hardware. 'Design' is equally as self-explanatory. It's the look and feel of your user interface or HMI. The interesting one is 'behavioral science', which is a purely human-focused discipline. "Behavioural sciences abstract empirical data to investigate the decision processes and communication strategies within and between organisms in a social system." What all this terminology means is that, more than ever, your apps need to be soundly-architected prepared for the complexity and unpredictable nature of human interaction. Let's look at what that entails.

Until just recently, most of us have been reliant on keyboards and mice in Flash, which are intermediaries between human hands and the computer screen. With touch technology, the middle step is removed and our hands are working where our eyes are looking. The term 'multi-touch' is perfectly self-describing: "multi-" + "touch". Many simultaneous points of contact or tactile input. It changes the dynamic for interaction design altogether. In my opinion, it's freeing in many ways. In the past, across the vast majority of Flash-based applications, there has been only one mouse cursor/point of interaction. It may seem overly-simplistic, but users can get accomplish much more in a shorter amount of time with multi-touch. Thus, the user experience is that even richer and more engaging. Even though the human finger is about 44 px wide on a 320x480 device, your app can help them get things done faster.

In some ways, the user (and developer) are thwarted by single-point interaction in Flash. For instance, let's think about a XBOX 360. Now imagine it with one single button on the controller. It just doesn't make sense in modern day devices. Even the original Nintendo had 4 buttons and a directional pad. That was over 20 years ago. Yes, it's possible to use combination input commands though the ActionScript Keyboard class and context menu. However, the thought process we want to start thinking in terms of, is the concept of "devices conforming to us." It's just not the same experience as one hand on a mouse, or 1-2 hand(s) on a keyboard. With directly tactile interactivity, there is countless complex processing of user input being captured and recalculated constantly. It's something altogether more sophisticated, which can lead to an infinitely more compelling user experience. That's enough theory for now, let's move on to the technical side of things.

One thing to keep in mind throughout this whole process, is that multi-touch enabled Flash applications have been around for years, but were used in conjunction with more low-level programming languages like C. In fact, you can find cached pages on Google from at least as far back as 2001 mentioning Flash and multi-touch together. Given the low-level partner languages, it's a good idea to know as much as you can about how Flash Player talks to hardware. Before I begin, I want to say that I am by NO means an expert on hardware or the protocols that link to software. I am purposely giving a high-level overview, because A) I couldn't possibly cover all of the documentation for each solution, B) I am still learning some of this myself, and C) technology changes all the time. So, if I miss something, or am incorrect, remember me saying this. :) Ok, disclaimer issued, on with the show.

There are two classes in the current Flash Player API that are gateways to hardware communication:

flash.net.Socket
flash.net.XMLSocket

The Flash Socket class "enables code to make socket connections and to read and write raw binary data," which basically means that Flash speaks 0s and 1s, the lowest language of any digital communication. If you are new to these APIs, check out Lee Brimelow's tutorial on Socket Fundamentals. With 3rd party APIs, the Socket connection depends on a Command Proxy Server, usually of the Java and C variety, but sometimes Python and Cocoa instances are used. These socket servers typically utitlize the TCP/IP or sometimes the UDP protocol.

There are a few socket proxy servers out there already for Flash, but FLOSC (Flash Open Sound Control) is one of the more popular choices. FLOSC uses the XMLSocket class to transfer information, which basically means that the data is typed to the XML format, is a bit slower, and "each XML message is a complete XML document, terminated by a zero (0) byte." FLOSC is a Java-based proxy server that receives events through a standard stateful touch-event protocol called TUIO (Tangible User Interface Object). "TUIO is an open framework that defines a common protocol and API for tangible multi-touch surfaces." As you'll quickly find out, you can't do much of anything in multi-touch without hearing the term 'TUIO'. It's the standard messaging event system for almost all touch-based hardware.

Ok, so you've got your hardware event communication, your proxy server, now all you need is an API to use with ActionScript. Enter Touchlib, an API from the NUI (Natural User Interface) Group, a massive, growing community in the world of multi-touch hardware and software communication. Note, however, that the Touchlib source code from Google Code hasn't been touched in awhile, and you should check the forums for any changes.

You should also know that Touchlib isn't the only API for multi-touch and ActionScript. In fact, there are several, though specific communication implementations vary. Again, read each one's documentation for full details. Your mileage may vary, professional driver on closed course, etc., etc..

Touche - A camera-based multi-touch tracking environment for MacOS X.

Ideo - A Multi-touch Package for Flash & Processing.

MapsTouch - A Google Maps multi-touch application written in Actionscript 3.

fiducialtuioas3 - A framework that allows you to use fiducial interaction in Actionscript 3.

Evoluce FMF - A Flash Multitouch Framework that integrates with the Evoluce Multitouch Input Management Software package.

GestureWorks - An extensible Gesture Library that includes pinch and scale, rotate, drag (one finger drag, two finger scroll), flip/flick, tap, double-tap, and press and hold.

Enough of the 3rd party stuff, whaddya say we crack open some of the newly native ActionScript!? Note that this is subject to change and may not be complete, but as Danny Dura showed during his MAX presentation, this code has already been tested and confirmed to work on both the iPhone and MacBooks that support multi-touch!

First the ui new package and inputMode events:

flash.ui.Multitouch
[static] inputMode:String
[static] maxTouchPoints:int
[static] supportedGestures:Vector<String>
[static] supportsGestureEvents:Boolean
[static] supportsTouchEvents:Boolean

flash.events.MultitouchInputMode
Multitouch.inputMode
MultitouchInputMode.TOUCH_POINT
MultitouchInputMode.NONE
MultitouchInputMode.GESTURE

// You MUST set this or your multi-touch code will not be registered 
// and therefore will NOT receive or broadcast events.

The new multi-touch event API is based on the ActionScript 3.0 Event model but is heavily influenced by the TUIO event API. In the Flash Player 10.1 API, much like the MouseEvent class has the following events:

MouseEvent.CLICK
MouseEvent.DOUBLE_CLICK
MouseEvent.MOUSE_DOWN
MouseEvent.MOUSE_UP
MouseEvent.MOUSE_MOVE
MouseEvent.MOUSE_OUT
MouseEvent.MOUSE_OVER
MouseEvent.ROLL_OUT
MouseEvent.ROLL_OVER

the flash.events.TouchEvent class has the following low-level events, which can be called from any DisplayObject:

TouchEvent.TOUCH_BEGIN
TouchEvent.TOUCH_MOVE
TouchEvent.TOUCH_END
TouchEvent.TOUCH_TAP
TouchEvent.TOUCH_OUT
TouchEvent.TOUCH_OVER
TouchEvent.TOUCH_ROLL_OUT
TouchEvent.TOUCH_ROLL_OVER
TouchEvent.TOUCH_TAP

Many multi-touch developers like to use an Object or Array of points to keep track of touch inputs (aka 'blobs') as well as an ID, and X/Y coordinatees for each point. It looks like Flash Player 10.1 will help you conform to these conventions, ActionScript-style.

flash.utils.Dictionary   // Used to track Arrays of Points
flash.geom.Point         // One contact point
TouchEvent.touchPointID  // Unique ID for each touch contact point
TouchEvent.stageX 
TouchEvent.stageY
TouchEvent.isPrimaryTouchPoint

We've been using single-touch point-of-sale (POS) interfaces for years, but modern multi-touch hardware technology has provided us with new and detailed interaction possibilities. Think about movie ticket kiosks, ATMs, and self-checkout lanes at the grocery store. They seem rather basic to us now when we think about them. Multi-touch brings the addition of gesturing. In Flash Player 10.1, GestureEvents are a high-level API package that extend the TouchEvent class. TouchEvents are much more low-level and segmented, but can be used to create your own unique Gesture classes. Gesture recognition adds subtlety and nuance to an otherwise, '0 or 1', 'on or off,' interactive system. Behind the scenes though, calculations are being continuously performed, as interpreting human gestures requires sophisticated mathematical algorithms. Though Apple didn't invent the concept. gesturing became a universally known device interaction behavior when the iPhone hit the streets, and has set the standard for touch behavior on devices. Users now expect gesture recognition to be present on even the most basic of touch-based applications. With Flash Player 10.1, gestures can finally be implemented, leaving the mouse and keyboard behind.

Gesturing is now handled by multiple classes in Flash Player 10.1. They're pretty straightforward:

flash.events.GestureEvent;
flash.events.GesturePhase;
flash.events.GesturePhaseEvent;
flash.events.TransformGestureEvent;

// GestureEvent 
GestureEvent.GESTURE_TWO_FINGER_TAP
GestureEvent.phase: String
GestureEvent.localX:Number
GestureEvent.localY:Number
GestureEvent.stageX:Number
GestureEvent.stageY:Number

// GesturePhase
GesturePhase.BEGIN
GesturePhase.END
GesturePhase.UPDATE

// GesturePhaseEvent 
GesturePhaseEvent.GESTURE_PAN
GesturePhaseEvent.GESTURE_PRESS_AND_TAP
GesturePhaseEvent.GESTURE_ROTATE
GesturePhaseEvent.GESTURE_ZOOM //Is the same thing as 'pinch'
GesturePhaseEvent.offsetX:Number
GesturePhaseEvent.offsetY:Number
GesturePhaseEvent.rotation:Number
GesturePhaseEvent.scaleX:Number
GesturePhaseEvent.scaleY:Number

Gesturing also marries perfectly with physics engines and another multi-touch mechanism made popular by iPhone: accelerometer support. I personally can't wait to use gesturing with ActionScript physics engine APIs like Box2DFlash and QuickBox2D. Altough indirectly related, accelerometer support in Flash coming, which means that gesturing and physics will become the a fundamental element of your mobile multi-touch applications (accelerometer support is currently only available on mobile platforms). Physics are to the accelerometer as water is to skiing; you have to have one to have the other. OK, so it turns out I'm not great with analogies, but here are some of the new Stage APIs for handling orientation recognition as well as accelerometer:

flash.display.StageOrientation; 
// e.g. StageOrientation.DEFAULT

flash.display.StageDisplayState;
//e.g. stage.displayState = FULL_SCREEN_INTERACTIVE;

Unfortunately, these are still partially under wraps, but I think this gives you a good idea of what to expect.

A vital resource that we as developers can lose sight of, is the role of our own imagination in designing an app. Imagination and ingenuity are what drives the most awe-inspiring experiences we've seen to date in Flash. This is especially true with touch-enabled applications. Thinking outside the box aids in improving existing application workflows and UIs as well.

Let's look at an example of re-visualizing an app, but using multi-touch input instead of a mouse and keyboard. Imagine a currently existing AIR application, but designed for multi-touch interaction. Let's use Adobe's Acrobat.com AIR app as a case study. Just think of the interaction possibilities within one view of the UI.

Think about the countless fluid and organic interfaces that have already been designed for today's Flash applications. One in particular that comes to mind, and one of my personal favorites when I first began developing in Flash, is selftitled.ca. This site is the "personal and commercial workspace" of Red Interactive's Sr. Interactive Developer/ Flash Platform Lead John Iacoviello. It's a site which could lend itself seamlessly to the future of mouseless interaction. Earlier, when I said that multi-touch interaction can be "freeing", sites like John's are exactly the kind of case where this could be true. Imagine John's site, but with the same interaction from something like this demo. When I see demos like that, it makes fluid Flash interfaces "make sense." It just seems like that's what the developer had in mind when originally designing the site. To bring these thoughts together, (and just because I think they're cool), here are some spectacular Flash-based video game sites that I think would be even more fun to use if they were redesigned to support multi-touch. (Remember: Use your imagination! You'll start to see the possibilities :) )

Little Big Planet's Sticker Room

Halo 3: Believe - Interactive Memorial

Let's take a step back a minute. It's 2009. Mouseless interaction technology is not just limited to mobile devices, touchscreen PCS or tablets. It's not even limited to game controllers like the WiiMote. What if, you didn't need any of those? Just because you're not using a mouse or keyboard, doesn't necessarily mean that you're touching something. A number of years ago, I saw some examples of wildly inventive interfaces that I will never forget. My Interaction Design hero, Brendan Dawes, is renowned for his experiments in Flash, Processing and other creative languages. Outdoing himself yet again, Brendan shows some demos of some tests he performed using play-doh as a video playback control, and as a piano playing mechanism. His description is one of a beautiful user experience.

"As I twist the Play-Doh and take bits away, the film reacts accordingly in real-time. Add too much Play-Doh and the film rapidly speeds up. An intimate connection is made between the user and the media. Every action has a reaction in the digital space. No scary buttons to press. No instructions to read. It’s just Play-Doh."

Simply unbelievable, but entirely possible. Think about it: it was a form of touch-based, reverse Augmented Reality. Mind-blowing, creative, and incredibly inspiring to any Flash developer.

Well, my months of research and study of Flash-based multi-touch tech are now in your hands. Go build some awesome multi-touch, (or no-touch) apps in Flash!

As an added bonus, I have accumulated a TON of links in my last few months of research. So many, in fact, that I have created an entirely separate link dump on my personal blog. Note that not all of the examples are Flash-based, but they're all certainly inspiring. There are over 100 links, so grab a Mountain Dew and some Fritos, kick back, and enjoy!

Read more from Kevin Suttle.