In Part I of “Design Patterns for Mobile Faceted Search,” I looked at the challenges and opportunities of mobile faceted search. To address the well-known challenge of limited screen real estate on mobile devices, I covered the Four Corners, Modal Overlay, Watermark, and Full-Page Refinement Options design patterns, which maximize the real estate available for search results on a mobile device. This month’s column covers strategies for making people more aware of the filtering options that are available to them, as well as methods of improving transitions between the various states a user encounters in a search user interface.

Teaser Mobile Design Pattern

Teasers are nothing new on mobile phones. Most mobile user interfaces implement this pattern in some manner or another. Consider, for example, the Yelp iPhone application, shown in Figure 1.

Figure 1—Teaser design pattern in Yelp’s mobile app

Yelp displays five complete search results, plus a teaser at the bottom of the page that shows a partial view of the next result set. The Teaser design pattern is very useful in showing that there are more search results below the fold. As I mentioned in a previous column, “Choosing the Right Search Results Page Layout: Make the Most of Your Width,” my usability testing has shown that Amazon is an absolute master of this pattern. Regardless of a screen’s resolution, they always display partial product descriptions and pictures just above the fold. Showing part of what is at the fold is a very effective means of inviting visitors to scroll down and see more content.

Recently, when Microsoft came out with Windows 7 Mobile, they extended the Teaser pattern well beyond just showing the availability of search results below the fold. In Figure 2, an excellent video from Luke Wroblewski’s blog entry “Windows Phone: User Interface Teases & Transitions” effectively demonstrates this concept. As you can see, Windows 7 Mobile shows there is more content available on the left, right, above, and below what’s currently visible on the screen.

Figure 2—Teasers and transitions in Windows 7 Mobile (Compiled by Luke Wroblewski)

In its implementation of the Teaser pattern, Windows 7 Mobile uses lots of very obvious cues, including screen titles that are cut off in midword, partially visible screen widgets, image fragments, and abundant use of animated transitions.

A teaser is a specific application of the more general design principle: fixing imperfection, in which a design immediately engages people by having them fix something that is intentionally not quite right with a user interface or object. In the process of fixing an imperfection, people learn the user interface’s interaction model. Studies have shown that this process of fixing imperfection, or seeking symmetry, is very natural to humans and quite immersive, even to the point of alleviating anxiety and pain in burn trauma.

The problem some may see with the Windows 7 Mobile design is that people cannot win at this particular game of fixing imperfection. There is absolutely no way to fix the user interface to simultaneously show all of a title, widget, or image that exceeds the size of the screen. It’s kind of like a blanket that is intentionally made too small: if you cover your legs, your chest is exposed; pull the blanket up to your chin and your legs get cold. As UX designers, we often seek a kind of minimalistic, authentic beauty and symmetry in our designs, so some designers might find this kind of user interface profoundly disturbing. However, one cannot argue against the effectiveness of this design approach.

The wireframe shown in Figure 3 takes advantage of the Teaser design pattern to show a partial view of some faceted search filters on the right and, thus, expand the available virtual screen real estate.

Figure 3—Teaser design pattern facilitates the discovery of faceted search filters

The search results are on the left in this wireframe; filters, on the right. Following a basic convention of mobile user interfaces, buttons on the right let people drill down deeper into the application’s information architecture (IA) space, while tapping buttons on the left moves them closer to the top of the IA or to the home page. In this example of what Microsoft calls panoramic applications, the mobile device acts as a sort of viewfinder that displays only a small part of a much larger virtual space.

The Teaser design pattern very effectively facilitates discovery through its use of partially exposed screen elements—in my example, faceted search results filters. This pattern also enables people to make rapid transitions from looking at search results to narrowing down the search results, so it is highly suitable for applications in which it is advantageous for people to discover a set of filters quickly and use them often.

Basic/Advanced Parallel Architecture Pattern

Another good design pattern for efficient discovery and use of filters is the Basic/Advanced Parallel Architecture design pattern, depicted in Figure 4. This design pattern is especially effective in applications that offer a fixed set of filters—for instance, booking air travel.

Figure 4—Basic/Advanced Parallel Architecture design pattern

The idea behind the Basic/Advanced Parallel Architecture design pattern is that there are two modes of interacting with filters: basic and advanced. The basic mode lets customers easily engage with your application and dramatically shortens their learning curve. However, people who want an advanced mode of filtering can easily obtain this functionality—in the example shown in Figure 4, by returning to the filter page and tapping the Advanced Search tab.

The parallel architecture part of this pattern comes in when a customer needs to adjust the filter values. For customers who started their query from the Basic Search tab, tapping Back returns them to that tab. For customers who started from the Advanced Search tab, tapping Back returns them to that tab, where they can adjust the expanded set of filters. While the relationships between these three contexts—Basic Search, Advanced Search, and Search Results—are a bit difficult to explain, they offer a very natural interaction model that most people have found very intuitive to use during testing.

The Basic/Advanced Parallel Architecture design pattern offers the added advantage of a flatter architecture. Instead of an advanced mode that requires people to go deeper into an application’s IA, the advanced flow sits in parallel with the basic flow, so both are equally accessible with a single tap.

It is important to note that, in both cases, the search results page has a persistent status bar across the top of the screen, displaying an entire search query, plus any applied filters. Recall that a persistent status bar was one of the features I described in Part I of “Design Patterns for Mobile Faceted Search.”

One application that successfully implements the Basic/Advanced Parallel Architecture design pattern is the Thirsty Pocket iPhone app, shown in Figure 5.

Figure 5—Basic/Advanced Parallel Architecture design pattern in Thirsty Pocket

Notice that, instead of labeling this feature Advanced, Thirsty Pocket uses the label More Search Options. In Thirsty Pocket, the purpose of the advanced search page is to provide options for searching outside the device’s current GPS location, as well as narrowing a search by price. Most people would not use the location filter, and the price filter might, in fact, severely limit the effectiveness of search by too frequently narrowing a search to zero search results. The price filter is a great example of the “ejector seat lever” Alan Cooper so eloquently describes in his book About Face 3. Therefore, this price filter is best hidden on a separate page. In the Thirsty Pocket app, the label More Search Options describes the purpose of the advanced search page perfectly and has the added bonus of being less intimidating.

In another variation on the Basic/Advanced Parallel Architecture design pattern, instead of Basic Search and Advanced Search tabs, Thirsty Pocket presents two buttons on its home page: a prominent Search Near Me button for basic search and a de-emphasized More Search Options button for advanced search options. Tapping More Search Options lets you set filters, then tap Advanced Search to see the filtered search results. Nevertheless, the Thirsty Pocket user interface still preserves the essential features of this pattern, because tapping Modify Search on the search results page takes you back to the appropriate search page—basic or advanced—depending on how you initiated a search.

In Conclusion

A lot of people have asked me about these design patterns and the best ways to use them. Thank you all for your questions. Unfortunately, the answer would go well beyond the scope of even a two-part article. The best recommendation I can make is that you learn as many design patterns as possible; understand why these patterns have emerged, what customer and business goals they support, and what limitations they have. Then, once you understand the unique goals of your own customers and business leaders, you can either choose the right design pattern or combination of patterns or develop your own design patterns that fit the task at hand. Our understanding of user interfaces is still evolving—and for mobile design, still in its infancy. Mobile design patterns and paradigms are just beginning to emerge, but they represent the beginning of something magnificent.

References

Bucolo, Sam et al. “The Design of a Tangible Interaction Device to Alleviate Anxiety and Pain in Pediatric Burns Patients.” CHI ’06 Extended Abstracts on Human Factors in Computing Systems, Montreal, Quebec, Canada, April 22–27, 2006.

Cooper, Alan, Robert Reimann, and David Cronin. About Face 3: The Essentials of Interaction Design. 3rd ed. Indianapolis, IN: John Wiley & Sons, Inc., 2007.

Nudelman, Greg. “Design Caffeine for Search and Browse UI.” Information Architecture Summit 2010, Phoenix, Arizona, April 9–11, 2010. Retrieved April 30, 2010.

—— “Design Patterns for Mobile Faceted Search.” UXmatters, March 8, 2010. Retrieved April 30, 2010.

—— “Experience Design for a Viral Mobile Community.” Net Squared Conference, San Jose, CA, May 2009. Retrieved April 30, 2010.

—— “Choosing the Right Search Results Page Layout: Make the Most of Your Width.” UXmatters, March 9, 2009. Retrieved April 30, 2010.

Wroblewski, Luke. “Windows Phone: User Interface Teases & Transitions.” LukeW Ideation + Design, February 17, 2010. Retrieved April 30, 2010.

In my previous Search Matters column, “Mobile Finding: Turning Limitations into Opportunity,” I discussed how mobile search user experiences differ from those on the Web. In this and my next column, I’ll look specifically at the challenges and opportunities of mobile faceted search. This column covers design patterns for maximizing the real estate available for search results, while the next will cover strategies for making people aware of filtering options.

Faceted search is extremely helpful for certain kinds of finding—particularly for ecommerce apps. Unfortunately, the designers of mobile applications do not have established user interface paradigms they can follow or abundant screen real estate for presenting facets and filters in a separate area on the left or at the top of a screen. To implement faceted search on mobile devices, we need to get creative rather than following established Web design patterns. Join me in exploring the Four Corners, Modal Overlay, Watermark, and Refinement Options design patterns for mobile devices. Following these patterns can move us one step closer to making faceted search a usable reality on mobile devices.

But first, let’s take a look at the challenges of designing mobile faceted search, which include navigational elements that use up precious screen real estate, limited search-refinement options, and the general lack of an iterative refinement flow.

Mobile Faceted Search Challenges

Recall that one of the mobile applications I discussed in my last column was Amazon’s iPhone app. Figure 1 shows its search results screen.

Figure 1—Faceted search in Amazon’s iPhone app

On the Web, Amazon’s faceted search is nothing short of gold standard, so comparing their Web faceted-search experience to that of their iPhone app is instructive. Looking at the Amazon iPhone app in Figure 1, the first thing to note is the amount of screen real estate Amazon has devoted to things other than search results. Their fairly standard iPhone page layout has a navigation bar at the top and a tab bar at the bottom, which together take up about 74 pixels. In all, Amazon has permanently devoted almost 22% of their precious screen real estate to navigation and other features. As a consequence, they can display only three search results at a time. And Amazon is actually fairly frugal with its screen real estate—at least in comparison to other mobile ecommerce applications.

Another interesting issue is that the user’s keyword query—in this case, green lantern—does not appear in its entirety in the application’s navigation bar—both because the user interface displays the query in a really large font and because the query text has to compete with a large search refinement button that is also on the navigation bar. With the screen‘s limited real estate, the query gets cut off, allowing only green… to appear there.

Last, but not least, the only search refinement Amazon presents to customers is By Category—a button that takes a customer to a different category-selection screen. While category is definitely one of the most useful and frequently applied facets, offering category selection alone hardly compares to the rich array of search-refinement options that is available on Amazon’s Web site, as shown in Figure 2. Conspicuously absent are various sorting options, as well as filtering by price.

Figure 2—Search on Amazon’s Web site presents many refinement options

However, in these screenshots, you cannot see one of the most important and striking differences between search on the Web and on mobile devices, which you can understand only in terms of the app’s search-refinement flow. Figure 3 illustrates the differences between Amazon’s Web and mobile search-refinement flows.

Figure 3—The differences between the Web and mobile search-refinement flows

On the Web, Amazon’s search supports progressive, faceted search refinement and exploration as one of its primary goals for the search process. As Peter Morville said in his brilliant book, Search Patterns, “Faceted navigation … helps us learn. Search becomes an iterative, interactive experience where what we find changes what we seek.” For example, on the Web, a customer who starts with the keyword query Nike can narrow down the search results by category (Shoes), additional keywords (Nike Air), product type (Cross Training), and size (12).

In contrast, Amazon’s iPhone app does not preserve search refinement as part of its finding flow. The customer who starts with the keyword query Nike, then refines the results By Category (Shoes) is unable to refine the results further. If a customer attempts to refine a query by adding more keywords (Nike Air), the app removes the customer’s prior category selection (Shoes), doing a completely new search for the query Nike Air and searching all categories. This behavior has the unfortunate effect of interrupting the flow of step-wise exploration and discovery that is central to the faceted search experience. Obviously, there is a real need for better patterns for the step-wise narrowing, expanding, and refining of faceted search results on mobile devices.

Mobile faceted search needs to balance customers’ needs to both refine results and maintain their search-refinement flows against the limited screen real estate and fat-finger issues. We must especially guard against introducing false simple-mindedness in the name of simplicity. Simplicity is a great goal, but as John Maeda said in his masterful book, The Laws of Simplicity, “Some things can never be made simple.” Using the words of Albert Einstein, we have to strive to make the most of the tools available to us and make user interfaces “as simple as possible, but not simpler.” The four design patterns that follow make the most of the available screen real estate, while providing intuitive and useful search-results refinement capabilities and improving the mobile faceted-search experience.

Four Corners and Modal Overlay Patterns

As I described in my column “Search Results Satori: Balancing Pogosticking and Page Relevance,” maximizing the number of results on a search results screen is key to improving information scent and overall finding efficiency. One of the most successful design patterns for maximizing the use of screen real estate is the Four Corners pattern, shown in Figure 4.

Figure 4—Four Corners and Modal Overlay patterns

The Four Corners pattern devotes almost the entire mobile screen to search results. Users can display particular functions by tapping the semitransparent corners, which provide access to filters, the home page, and two additional menus. This pattern includes a thin, persistent status bar across the top of the screen, which displays an entire search query, plus any applied filters, rather than just the few characters Amazon Mobile displays in a large font. Displaying an entire query is highly beneficial, given users’ mobile context of use and the frequent interruptions inherent in the mobile finding experience.

Although the design pattern is called Four Corners and triangles are the most effective shape for the corners, they don’t necessarily have to be triangles. Other alternatives include semitransparent buttons or floating icons. However, for transparent buttons, it’s best to keep icons simple and avoid text labels.

Another pattern that works really well with Four Corners is the Modal Overlay pattern, shown on the right in Figure 4. A modal overlay can display various search-refinement options such as filtering and sorting. As I wrote in “The Mystery of Filtering by Sorting,” it is not necessary to place filtering and sorting in two different areas of a screen. Most people have, at best, just a vague idea of the differences between the two features. This is particularly true for the mobile experience, where screen real estate is at a premium and minimizing the number of taps necessary to achieving a goal is key.

The Four Corners and Modal Overlay patterns work particularly well in combination. In Figure 4, each user interaction on the modal overlay refines the search results interactively. This dynamic interaction model works great, because users can always see where they are and what filters they have applied, and they have easy access to further refinement options, all without ever leaving the context of the search results screen.

Some touch phones—like the iPhone—do not provide an API (Application Programming Interface) for modal overlays, but instead deliver content one screen at a time, with transitions between screens. On such devices, you’ll need to fake modal overlays. You can achieve this effect by rendering the same search results on a new screen—this time with a slightly darkened background—with a modal overlay over the results. When displaying this new screen, an application should minimize any transition to help to preserve the illusion of maintaining the search results context.

Watermark Pattern with the Full-Page Refinement Options Pattern

The latest generation of smartphones—including the Apple iPhone, Palm Pre, and Google Android—provides designers with a whole bevy of novel interaction models. Some of the ways in which people must interact with new smartphones may not be obvious to them, because they may not match their existing mental models or the affordances of devices with which they’re familiar. Nevertheless, once people discover these interactions, they seem surprisingly intuitive and fun, and people do not easily forget them.

In mobile computer games, it is common practice to teach players by allowing them to manipulate their mobile devices through various gestures, including tilting, turning, and shaking them—or even using a device’s accelerometer to mimic the way one would use a magic wand: “Incendio!”

Figure 5—Fun ways of employing the iPhone’s accelerometer in the Harry Potter game

In ecommerce applications, customers’ object in interacting with a mobile device is not typically entertainment, but the efficient acquisition of goods, services, or information—so necessity typically shortens their learning curve. If we want people to be able to perform a nonstandard interaction to use a device’s essential features, the interaction must be fairly obvious—and it must not require them to read instructions they would perceive as obnoxious or distracting. This is where careful use of the Watermark design pattern can be especially powerful.

A watermark is a transparent outline that appears either over or behind the primary content on a screen. In some ways, a watermark is similar to a splash screen, but it is considerably less distracting because watermarks let users see the content on the screen at the same time. Figure 6 shows two variations of the Watermark pattern, plus a full-screen example of a Refinement Options pattern.

Figure 6—Two variations of the Watermark pattern and a Refinement Options pattern

The two examples of the Watermark pattern, shown in Figure 6, demonstrate a circular swiping gesture and a shaking gesture on a multitouch device. In both cases, once a user launches an application, each watermark appears just once or twice, gently dissolving to let the user view the screen’s main content. Watermarks can also use animation—for example, to demonstrate the motion of the circular swipe by displaying one arrow at a time on the screen. However, animation greatly increases a watermark’s visibility, so you should use it with care. Circular swiping emulates the motion of digging into a pile of puzzle pieces to find the right one, while shaking recalls the action of shaking up small, loose objects in a bag or a box.

In each case, the idea behind the watermark is to educate or remind people about the gesture or interaction that is necessary to view a screen containing additional search-refinement features. Once people decide to dig in further or shake up their search results, the application displays a full screen of refinement options, as shown at the bottom of Figure 6. This screen provides an example of the Refinement Options pattern. Again, the thin, persistent status bar across the top of the screen shows the query. Together with the watermark, this helps maximize the real estate the screen can devote to search results, while making exploration intuitive.

With a full screen of refinement options, a user can view four or five of the most popular values for each of the facets and select a facet value with a single tap. This is almost like having the entire array of refinement options on a typical faceted search results Web page. Having the search box integrated into the refinement options page also re-enforces the idea of applying search query terms and facets together as a set, just as they are on a Web page. Together, these patterns let us duplicate the intuitive and efficient step-wise refinement and expansion flow our customers expect from a faceted search results page on an ecommerce site. Using a scrollable screen of facets rather than a modal overlay like that shown in Figure 4 allows us to devote the full screen width to the search results, sprucing them up with icons for popular refinement options and letting customers refine the results with fewer taps.

Does this mean presenting a full page of refinement options is better than a modal overlay? Not at all. A modal overlay displays its refinement options within the context of the search results, while displaying a separate results-refinement screen, along with its associated transition, yanks customers out of the search-results context and presents what some people might think is an overwhelming variety of refinement options.

When discussing design patterns, avoid getting caught up in judging one pattern to be better than another. Each pattern is useful in specific contexts and applications and for particular purposes and types of customers. Design patterns are like a language we can use to communicate with our customers. It helps to have a vocabulary of patterns that is as complete and varied as possible, as well as to use each pattern appropriately to communicate your design message with clarity and precision.

Thinking of porting your Web finding experience to iPhone, Android, or Windows Mobile? Just forget about the fact that these devices are basically full-featured computers with tiny screens. Having gone through this design exercise a few times, I have realized that designing a great mobile finding experience requires a way of thinking that is quite different from our typical approach to designing search for Web or desktop applications. To put it simply, designing a mobile finding experience requires thinking in terms of turning limitations into opportunities. In this column, I’ll discuss some of the limitations of mobile platforms, as well as the opportunities they afford, and share a few design ideas that might come in handy for your own projects.

Understanding Mobile Platforms

One of the challenges of mobile application design is understanding both the capabilities and limitations of each platform. Let’s use the iPhone finding experience as an example. On the plus side, the iPhone has a high-resolution screen, Multi-Touch controls, accelerometer, persistent data storage, cool video transitions, push content delivery, GPS, and a device ID. The benefits of these features have been pretty much beaten to death in advertisements, so I will not discuss them here. On the other hand, the problem constraints and limitations of mobile devices are much more interesting. I have found few sources that discuss these in detail, so in this column, I’ll attempt to describe the most important challenges of designing for the new generation of smartphones—at least as they pertain to finding:

• the difficulty of typing
• the small amount of screen real estate
• awkward touch controls
• the so-called fat-finger problem

The Difficulty of Typing

Searching requires users to type a search string, and typing on a mobile phone is difficult—partly because of the size of the device. In a July 2009 blog post on Alertbox, Jakob Nielsen called the mobile experience “miserable” and reported, “Text entry is particularly slow and littered with typos, even on devices with dedicated mini-keyboards.”

For many users, touch devices like the iPhone exacerbate this problem. Their screens display a virtual keyboard and other buttons. One issue with touch devices is poor visibility, because a user’s fingers must, by necessity, cover buttons on the screen when a user is pushing them. Thus, when pushing a button on a virtual keyboard, a user’s hand obscures his view of the keyboard.

The lack of tactile response on touch devices presents a particular problem when a user is multitasking—whether riding in taxi, using public transportation, or walking down a city street. On any mobile device, it’s very difficult to type when a person is crammed into a bus or subway car and being jostled around. A user’s desire to avoid typing becomes a necessity when using a mobile device while driving.

Another challenge users encounter when searching on smartphones is that they’re likely to lose anything they’ve laboriously typed into a search box if a device receives an incoming phone call, because mobile applications cannot block phone functions. Nor should an application, however useful, be able to prevent a user from receiving incoming calls.

The Small Amount of Screen Real Estate

Mobile screens are, by necessity, small, because a mobile device has to fit into a person’s pocket or purse. The small size of mobile screens limits the number of controls and the amount of content that can appear on them. In that Alertbox post I mentioned earlier, Jakob Nielsen reported, “Unsurprisingly, the bigger the screen, the better the user experience.” According to Nielsen, users’ success rates with touch phones like the iPhone are about double their success rates with feature phones.

Awkward Touch Controls

One of the consequences of mobile devices’ having smaller screens and controls that users must manipulate through touch interfaces is that some controls no longer look like their Web and desktop counterparts. For example, rather than the usual drop-down list or set of option buttons, the selection control on an iPhone is instead a spinning control called a picker, shown in Figure 1.

Figure 1—Three pickers in the Urban Spoon iPhone app

A picker that a user can quickly manipulate with a finger eats up precious real estate, so it’s not possible to show users much outside the picker. Large pickers also place limitations on how users can filter search results on mobile devices. Similar size restrictions exist for all of an application’s touch controls.

The Fat-Finger Problem

The high-resolution screens on better mobile devices—like the iPhone, Android, and Palm Pre—can accommodate fairly high information density. Unfortunately, at the same time, touchscreens limit the on-screen density of controls that users can accurately manipulate with a finger. Thus, placing multiple controls close together on a touchscreen mobile device presents difficulties. This challenge is known as the fat-finger problem.

Typically, an iPhone touchscreen can comfortably support a maximum of only three to five clickable buttons or tabs across a screen. More than this leads to frequent frustrations due to users’ inadvertently pressing the wrong control. While five controls across a screen is the absolute limit for relatively frustration-free mobile computing for people with relatively small fingers, I highly recommend a limit of four or fewer controls. Of course, if one control is bigger than the others, you’ll have to reduce the overall number of controls accordingly. For example, Figure 2 shows the Yelp mobile application, which has only three controls across the navigation bar at top of the screen, but four controls across tab bar at the bottom.

Figure 2—Yelp iPhone app

Because Yelp’s search box takes up a large part of the navigation bar, the designers had to reduce the overall number of controls to three to ensure users’ wouldn’t press the wrong controls with their fingers too often.

Understanding User Experience within a Mobile Context of Use

The challenge in designing mobile applications is the need to accommodate the design constraints and usability challenges mobile devices impose, while focusing on users’ goals within a mobile context of use. Simply duplicating the functionality of a Web application—while trying to work around the mobile design challenges I’ve described—always results in a subpar mobile application. It’s not enough to think: How can I duplicate our Web application’s user experience within the limitations of the mobile platform? Instead, it’s better to start from scratch, focusing on What experience would work best for mobile users? Putting users’ goals first allows a design team to concentrate on the new opportunities a mobile application presents rather than seeing the challenges of mobile simply as barriers to implementing a Web application’s existing functionality.

Next, I’ll present some ideas about how to approach the design of finding experiences for mobile devices in a way that lends itself to taking full advantage of their capabilities within a mobile context of use. These ideas by no means represent an exhaustive catalog of all possibilities. I merely want to provide a few examples that may inspire further exploration as part of your own finding projects.

Preloading Pertinent Search Results

As I discussed earlier, typing on mobile devices is difficult. Plus, a phone call, a text message, or an opportunity to take a picture is likely to interrupt a user’s finding experience at least once. Saving a user’s previous searches is an obvious and simple way of re-engaging users in a finding task and provides useful context when an application first opens.

Unlike Web applications, native mobile applications are persistent, so it’s easy to cache their search results. Cached results load quickly, users’ re-engagement is immediate, and there is little to compete for a user’s attention—that is, at least until another phone call comes in. Some mobile device APIs even enable native applications to detect whether a phone call interrupted a user’s previous session or the user exited an application normally and determine how much time has elapsed since a user last opened the application. These capabilities present interesting possibilities for fine-tuning an application’s welcome-back screen to re-immerse users in their previous tasks or offer pertinent new content and present new possibilities for interaction.

Providing Local Results

On mobile devices that support GPS, Wi-Fi, or other location-tracking mechanisms, you can determine the current location of another person who is using a mobile device, allowing applications to offer location-aware services. Mobile applications with search capabilities can serve highly relevant, fresh results that perfectly match a user’s current mobile context. For example, Loopt is just one of a whole cadre of social networking mobile applications that allow users to track the locations of their friends who are currently nearby and exchange messages with them, get coupons from local merchants, and discover neighborhood happenings, as Figure 3 shows.

Figure 3—Local search results in the Loopt iPhone app

Offering a Value-Added Interpretation of the Real World

Using mobile devices for sense-making in the real world offers one of the most intriguing possibilities for mobile applications. Luke Wroblewski described some interesting possibilities, including augmented reality applications, in his Smashing Magazine article “Enhancing User Interaction with First Person User Interface.”

When it comes to finding, however, the Amazon Mobile iPhone app offers the Amazon Remembers feature, which is a forerunner of many exciting things to come. Amazon Mobile lets customers take pictures of any real world items and add them to their lists of things to remember. Once a user uploads a photo, Amazon figures out what the item is and, if there is a corresponding item available for sale on Amazon, displays that item and sends the customer an email alert, encouraging her to purchase the item. Customers often get a response in a matter of minutes, but the search can take up to 24 hours. This application lays a solid foundation for the idea of a mobile Internet of objects that I described in one of my previous columns, “Brave New World of Visual Browsing.”

Providing Various Sorting Options

As I previously mentioned in my column “The Mystery of Filtering by Sorting,” sorting options are an excellent way of opening up an ecommerce site’s inventory for browsing. One of the best ways of doing this is to provide two or more buttons on a search results screen that allow multiple ways of dissecting an inventory, without ever failing to serve some results. By using sorting options together with geolocation, customers can even avoid having to type in any query at all. As Figure 4 shows, the ThirstyPocket iPhone app lets customers simply press the Search Nearest or Search Newest button to see a sample of local results without having to type anything in the search box.

Figure 4—ThirstyPocket iPhone app

Of course, a customer can always type keywords in a search box to narrow down the results. As I explained in my presentation at the Net Squared Conference in May 2009, using this design pattern lets customers engage with an inventory of items or content immediately, then invest the effort of typing in keywords once they have caught the scent of something that interests them or to refine the search results further. Of course, given the fat-finger problem and a mobile device’s limited screen real estate, we can’t provide more than three to five sort options on the screen at one time. However, as the ThirstyPocket example shows, even a couple of sort options is often enough for customers to begin exploring.

Considering Custom Controls

One consequence of mobile devices’ having a smaller screen is not having the space to create a navigation bar of filters, facets, or categories on the left, providing a key to the properties by which a user can effectively narrow down a result set. Consider, for example, the all-important category filter. If an application were to use the standard iPhone picker, shown in Figure 1, it would obscure most of the search results on the screen. Various applications deal with this challenge in different ways. Some mobile applications have created custom category-filter controls—like those in Amazon Mobile, shown in Figure 5.

Figure 5—Custom category-filter controls in Amazon Mobile

Clicking By Category takes a customer to a different, full-sized screen on which he can select a category. The ThirstyPocket application, shown in Figure 4, uses the same principle for its custom category selector, with an added twist: the visual design of the category selector is reminiscent of the familiar drop-down list, taking full advantage of customers’ existing mental model and helping them understand what behavior to expect.

Changing Search Paradigms

Because of the unique mix of constraints and opportunities that mobile application design presents, this design space is rich with possibilities for changing the existing paradigms for search and finding. Consider speech recognition, for example. While, on the desktop, speech recognition does not yet enjoy widespread popularity and use, mobile represents an entirely different context—where speech recognition can offer an ideal solution. Not interpreting a spoken word correctly on a mobile device might not be quite as big a deal as it is on the desktop, because the accuracy of speech recognition may actually approach, if not exceed, that of typing on a mobile phone’s awkward mini-keyboard. Combine speech recognition with the use of an accelerometer and magnetometer, allowing gestural input, and you have the Google Mobile search application for the iPhone, shown in Figures 6 and 7.

Figure 6—Google Mobile iPhone app

Figure 7—Google Mobile Voice Search on the iPhone

The iPhone application Google Mobile recognizes the gesture of a person’s swinging the phone up to his ear to know when to record a search command. When the user speaks, the search engine accepts and interprets his voice commands, then serves up search results. This user interface implements what is literally a game-changing design paradigm, because its designers have taken the time to truly consider the mobile context of use and map natural interactions like speech and gestures to mobile device functions. As Peter Morville said in his book Search Patterns, “We simply raise our phones to our ears and speak our search, relying on Google Mobile to derive what we want from who we are, where we stand, and what we say…. Like placing your hands under a tap to turn on the water, this is the type of smart design that ‘dissolves in behavior’.”

When it comes to designing for a mobile context, we are just starting to scratch the surface. As Tom Chi of OK/Cancel famously quipped in his interview with Luke Wroblewski, “A well defined-and exciting problem (and its associated constraints) is the catalyst that makes design go.” When we stop thinking about the limitations of mobile platforms and, instead, truly focus on the user goals we are working to support, we might just find those limitations turning into opportunities for redefining how people find, remember, and discover things in their world.

References

Morville, Peter. Search Patterns. Sebastopol, CA: O’Reilly, 2010.

Nielsen, Jakob. “Mobile Usability.” Alertbox, July 20, 2009. Retrieved February 27, 2010.

Wroblewski, Luke. “Enhancing User Interaction With First Person User Interface.” Smashing Magazine, September 21, 2009. Retrieved February 27, 2010.

Wroblewski, Luke. “Defining the Problem: Q&A with Tom Chi.” Functioning Form, April 27, 2006. Retrieved February 27, 2010.

See the presentation on Slideshare ⇒

ThirstyPocket used iPhone application design to foster social change by creating a platform for viral neighborhood commerce, which brings people together and encourages community interaction.  We will demonstrate buying and selling flows and discuss specific user experience design strategies and insights that were used to improve upon existing e-commerce offerings.

When the Web began, pages were mostly text, but today, everywhere we look, it seems like image content is taking over the Web. The ubiquitous use of digital cameras and improvements in the picture quality of mobile phone cameras has likely helped this phenomenon along. The shift toward content that is primarily visual introduces new challenges and opportunities for developing intuitive and powerful user interfaces for browsing, searching, and filtering visual content. To help me cover this important topic, I’ve asked Ahmed Riaz—Interaction Designer at eBay and physical interaction design enthusiast—to contribute his insights and ideas.

Introducing Visual Browsing

What, you may ask, is visual browsing? Loosely defined, visual browsing user interfaces are those that let people navigate visual content—that is, search for content using pictures. We will discuss four types of visual browsing:

1. browsing images or photos using text attributes or tags—for example, on Google Images and Flickr
2. using images in queries to describe attributes that are hard to describe with text—as on Like.com and Zappos
3. using images to facilitate wayfinding and navigation in the real world—such as on Google Maps and Photosynth
4. collecting and browsing images on mobile devices—like on RedLaser and NeoReader

Each of these approaches to visual browsing supports different goals for searchers and introduces its own unique challenges, design directions, and best practices, which we’ll explore next.

Browsing Images Using Text Attributes or Tags

While there are many image-centric sites, the majority of the Web’s image content is on either Google Images or Flickr. No discussion of visual browsing would be complete without mentioning these two sites. When it comes to finding images that are published somewhere on the Web, Google Images is the site most people go to. Google uses a proprietary algorithm to assign keywords describing each image, then index and rank all of the images in their enormous database. They also capture a thumbnail of each image and store it along with the source URL.

While Google Images is undeniably a very useful site, it exemplifies some of the challenges of automated image finding. The primary mechanism for finding images is a text-only query, which perfectly demonstrates the impedance mismatch that exists between text and image content. Because no single keyword can adequately describe all of the pixels that make up an image—and, most of the time, the keywords that surround an image on a page provide an incomplete description at best—Google Images interprets search queries very loosely. Typically, it interprets multiple keywords using an OR instead of the usual and expected AND operator. As a consequence, it omits some keywords from the query without letting the searcher know explicitly what is happening.

On the one hand, because Google Images interprets search queries in a such a loose fashion, it rarely returns zero search results. On the other hand, image searches sometimes give us strange and unpredictable results, while making it challenging to understand what went wrong with our queries and how to adjust them to find the images we’re really looking for. Figure 1 shows an example—the search results for Africa Thailand elbow Indian.

Figure 1—Google Images search results for Africa Thailand elbow Indian

Judging from the search results, Google Images seems to have omitted keywords randomly from the query, without indicating to the searcher it has done so. Of course, one could argue that getting such poor results for a query is an example of garbage in/garbage out. However, we think this example demonstrates that the keywords Google takes from the surrounding content on a Web page—and possibly some Web pages that link to it—do not always match an image. While an image may technically be located at the intersection of Africa, Thailand, elbow, and Indian, this by no means guarantees that the image would actually show Indian warriors demonstrating the perfect Mai Thai elbow-strike technique on the plains of the Serengeti.

Plus, a computer-generated index usually makes no distinction between the different possible meanings of an image’s surrounding keywords—for example, a search engine can’t tell when an image appears in juxtaposition to text to add humor. Nor do automated indexing engines handle images well when they have purposes other than providing content—such as images that add visual appeal or bling or those for marketing or advertising. Such image content is, at best, orthogonal to its surrounding text.

Image search filters that are based on text are, at the moment, not particularly robust. For example, Figure 2 shows Google Images search results for iPhone with some filters applied. As you can see, the resulting images have little to do with the iPhone.

Figure 2—Google Images filtered search results for iPhone

As a photo-hosting site, Flickr enjoys the distinct advantage of having people rather than computers look at, interpret, and tag the images they create and view. Users can search for images by their tags. While Flickr is not a perfect image-searching system, it provides an entirely subjective, human-driven method of interpreting and describing images. Why is that valuable?

According to Jung’s conception of the collective unconscious, we can assume all individuals experience the same universal, archetypal patterns. These archetypes influence our innate psychic predispositions and aptitudes, as well as our basic patterns of human behavior and responses to life experience. Likewise, at the root of all folksonomies is the inherent assumption that people—when we look at them collectively—tend to respond in similar ways when presented with the same stimuli. Simply put, people looking for images of cats would be quite happy to find the images that a multitude of other people have taken the time to tag with the word cat.

As Gene Smith describes in his book, Tagging: People-Powered Metadata for the Social Web, folksonomies and tagging usually work quite well—in most cases, nicely resolving the kinds of issues that occur in Google Images due to auto-indexing, which we described earlier. Unfortunately, the human element also introduces its own quirks along the way. Take a look at the example of a Flickr image shown in Figure 3.

Figure 3—One of the most interesting Flicker images tagged with iPhone

Among almost 2 million Flickr images tagged iPhone, Flickr considers this image one of the most interesting, with 785 favorites and 37 comments. As you can see at the bottom of the page, one of the problems with having a community-driven popularity algorithm is that some people post nonsense comments just to be able to say they’ve added a comment to a popular item.

Another problem has to do with tags that are strictly personal. For example, tags like GOAL:EXPLORE! and hawaalrayyanfav have meaning only to the person who actually tagged the item—and little or nothing to the population at large. On the other hand, adding tags like funny and humor is actually quite useful, because computers don’t recognize humor at all. In fact, humor is the aspect of image description where algorithm-based search results fail most spectacularly. We’ll talk more about folksonomies, controlled vocabularies, and searching for tags in future columns. For now, we’ll just say that neither way of finding images—through text tags or associated surrounding text—is perfect, and there is a great deal of room for improvement, perhaps through combining both search approaches, using a single algorithm and a user interface that does not yet exist.

Using Images in Queries

We think one of the most intriguing aspects of visual browsing user interfaces has to do with the mismatch between images and the text algorithms or people use to describe them. They highlight the inherent limitations of finding and managing images through text-based queries. Typically, a person sees an object, determines what to call it, then tries a few keyword searches based on that interpretation. Adding a photo to a query lets a searcher bypass the interpretation step and just let the computer see what the query is about.

On the Web, one of the more successful examples of image-based search is the shopping site Like.com, which lets searchers use images as part of their queries to describe the attributes of items that are difficult, if not impossible, to describe precisely using text alone.

While Like.com has some issues when it comes to presenting good entry points into their enormous inventory for people who like to browse, once you find an item you’re interested in, the search engine is absolutely phenomenal at picking up visually similar items. For example, Figure 4 shows the results for an image-based search that is based on a picture of a striped shirt. The algorithm is smart enough to explore useful and popular degrees of freedom, deviating from the starting image, while keeping the zeitgeist of the original image intact.

Figure 4—Like.com image-based search results

In addition to doing a great job of exploring the possible degrees of freedom from the original image, the system lets shoppers fine-tune their queries to match a specific part of an image by zooming in on the part they’re interested in. This is very similar to selecting part of an image on Flicker and tagging only that part of the image. For example, a user could tag a person in an image as friend and a snake in the image as giant anaconda, while naming the entire image The last-known picture of Billy—hopefully adding the tag humor, while he is at it. As images become more ubiquitous on the Web—and generate evermore layers of meaning—being able to select and annotate only part of an image will become more and more important in a visual browsing user interface.

The Like.com search algorithm can relax different attributes, including color, pattern, and shape, as well combine image searching with text attributes, demonstrating the potential of image-based searching and browsing.

While government sources can neither confirm nor deny this, the Department of Homeland Security is allegedly interested in image-based search technologies, mainly in regard to facial recognition. However, as Munjal Shah, CEO and Cofounder of Riya told the audience at the MIT/Stanford Venture Lab’s Next Generation Search Symposium in 2007, commercial-grade facial recognition technology is not quite there yet. Such search algorithms can generally recognize which part of an image is a human face and venture a fairly accurate guess at a person’s gender, but recognizing a specific face requires matching the precise expression and angle of the face in a photograph with the sample, which can often be difficult. Despite these limitations, the potential of image-based search technology is enormous.

Does a user interface need to have a true visual-search algorithm to perform image-based search? Although technology helps, it does not afford the only way of performing a visual browse. One alternative is to have a human being tag items, using abstract visual attributes—such as a range of icons or outlines of shapes, using limited screen real estate—that, in turn, would let customers describe their visual queries. This kind of a pseudo-image-based search could, for example, successfully complement the existing people who shopped for X, also shopped for Y metrics-based algorithm.

Using Images for Navigating in the Real World

By now, most people are familiar with street view in Google Maps, which is pictured in Figure 5.

Figure 5—Google Maps street view

Using this phenomenal technology is almost like being there, with virtual-reality controls for navigating through nearly continuous photos of real space. Google Maps is an excellent example of automated sense-making of images, using precise GPS coordinates. Visual browsing controls let users walk, turn around, and jump through real space, using virtual projection. However, as anyone who has looked up their street and their house can tell you, these pictures are frozen in time. Some people can even tell you when the Google Maps picture of their house was taken, because a friend’s car was captured in the satellite photo.

In contrast—or perhaps as a complement to Google Maps—Photosynth, Microsoft’s new acquisition, uses crowd-sourced photographs to construct and let users navigate through 3D street-level space. It can also add the fourth dimension of navigating through time, letting users cross-reference multiple images. The feel of the interaction evokes looking into vignettes and being able to shift your point of view through space and time. One of the best examples available of this new capability is the Photosynth of the Obama inauguration shown in Figure 6.

Figure 6—Photosynth of President Obama’s inauguration

What is really interesting about this emerging technology is that it sidesteps—quite literally—the issue of a finite density of information within a single photograph. With systems like this, we are moving into a world where—rather than relying on just keywords and tags—we can use deep image analysis to sort and restructure sets of images and make sense of what we are looking at. The computer can truly see a real 3D space.

Collecting and Browsing Images on Mobile Devices

Using images as input for searches on a hand-held device is reminiscent of using barcode readers to enter inventory counts into a computer. Historically, barcode recognition required the use of a specialized device that was available only to governments and larger businesses. However, recent developments in mobile technology are rapidly changing this paradigm. An increasing number of mobile phone applications like RedLaser, LifeScan, and NeoReader, which is shown in Figure 7, now let customers snap photos of barcodes, using their mobile phones, then immediately use the photos as input for a search algorithm.

Figure 7—NeoReader iPhone application

Another tool called Semapedia offers an interesting way of using mobile barcode recognition. Semapedia makes it possible to tag any item in the physical world with a barcode that contains a link to the appropriate Wikipedia Web page. A person can use a mobile phone with the free NeoReader application to read the link’s URL and navigate to the Web page providing Wikipedia content about the object or place a person is looking at. Figure 8 shows a historical site that is tagged with a barcode linking it to a Wikipedia page.

Figure 8—Historical site tagged with a Semapedia barcode

Another example of the capability of mobile barcode-recognition technology comes from the Suntory Company in Japan, which tagged their beer cans with a special 2D barcode. Scanning this barcode with the application NeoReader takes a consumer to a mobile Web site that lets visitors register to offset 100g of CO2 emissions once per day and get tips on mitigating their own greenhouse gas emissions.

Both of these examples demonstrate that we are moving ever faster toward a world that is populated by smart objects, which Bruce Sterling dubbed spimes. We can track spimes’ history of use and interact with them through the mesh of real and virtual worlds that pervasive RFID and GPS tracking create. Mobile picture search is certainly emerging as the input device of choice for connecting the real and the virtual worlds to create the Internet of Objects.

Peter Morville, in his book Ambient Findability, talks about the sensory overload, trust issues, and bad decisions that are sure to result from our interactions with the Internet of Objects. However, it is hard to elude the siren’s call of such technology. It is now almost possible to use technology similar to that of Like.com to analyze every image and frame of video a mobile phone captures and tag them with text, GPS coordinates, time, and author. At the same time, it is possible to cross-reference each image with other images of the same place or similar things, along with all the tags, content, and links an entire world of users has added to this collection of images. The technology to collect all of these images in a massive 3D collage that users can navigate along a time axis, Photosynth-style, is also just around the corner.

Interactions of this kind make it possible to establish a unique and natural connection between the real world and the Internet computing cloud. The key is the added meaning the computer provides, based on its understanding and interpretation of the content of an image. Imagine, for example, taking a picture of a CD cover with your mobile phone’s camera and sharing it with your friends to learn what they think about the music; using the same picture to get more information about the artist on the Web—for example, pictures, videos of recent concerts, and a biography; finding out where you can buy the CD at the best price online; or using GPS to find coupons to purchase the CD from a local merchant. Imagine being able to do all of this with only a few screen taps and without typing a single character, while listening to a sampling of the CD’s tracks, playing through headphones connected to your mobile phone.

We are just starting to scratch the surface of these types of massively distributed, image-based, human-to-mobile interactions. Although image-processing speeds are getting much better—witness the smooth, fast scrolling of iPhone photo album images versus the slow shuffle of images on a Treo—the time it takes to process images on a mobile device and send them back and forth between that device and a server is still the biggest barrier to this type of technology. Once the image feedback loop is nearly instantaneous, this new paradigm for mobile input will truly come into its own, bringing the digital world ever closer to the real world and ushering in the arrival of a brave new world of visual browsing.

References

Jung, Carl Gustav. Man and His Symbols. New York: Dell, 1968.

Morville, Peter. Ambient Findability. Sebastopol, California: O’Reilly, 2005.

Nudelman, Greg. “Making $10,000 a Pixel: Optimizing Thumbnail Images in Search Results.” UXmatters, May 11, 2009. Retrieved July 26, 2009.

Riflet, Guillaume. “Semapedia, or How to Link the Real World to Wikipedia.” Webtop Mania, August 26, 2008. Retrieved July 26, 2009.

Smith, Gene. Tagging: People-Powered Metadata for the Social Web. Berkeley: Peachpit Press, 2008.

Smolski, Roger. “Beer, Carbon Offset, and a QR Code Campaign.” 2d code, July 27, 2009. Retrieved July 26, 2009.

Sterling, Bruce. Shaping Things. Cambridge, Massachusetts: MIT Press, 2005.