Evangelizing A Product Concept By Validating A Design

The following article was first published June 7th, 2000. It describes a project to create a design for a set of wireless data applications for Nokia at their Americas office in Las Colinas, TX. This edition has been slightly edited for style, and in content to keep it relevant for an audience 16 years later. The original did not contain the name of the client, the team members or the narrative of the project, in order to protect client confidentiality. Given that almost 16 years have passed and the client is no longer in business, I feel it is reasonable to elaborate on the original article with the narrative detail.

At the time, I thought of what we were doing as Agile UX, because the design approach is iterative and rapid. Actually, we didn’t have the “Agile” word until 2001. The style of this process felt like eXtreme Programming for UX. It didn’t occur to me to think of it as Lean UX. The client was Nokia, at the time, the world’s largest mobile phone company. They were extremely secretive about their future plans. This wasn’t a startup even though the project was a “startup” within Nokia and the domain – wireless data applications – was bleeding edge at the time. The technology was WAP and our purpose was not only to create a business Nokia could exploit but to enable Nokia to use our work as an archetype to encourage network operators and 3rd parties to develop usable applications on WAP. [For those not familiar with WAP, it was a 2G wireless data technology which used a text-based screen format, via the hard key interface of what were known as “candy bar” phones.] read more…

Evangelizing A Product Concept By Validating A Design

In larger technology companies it can often be difficult to develop an understanding of the advantages of doing good product design early. This can be even more so with UX design processes that should be done early, close to the beginning of a project, while the product is being defined and the requirements written. At this stage it is often true there is no funding to build prototypes, to do testing, and with recognizable brands, it often isn’t acceptable to test something in the market. It is not unusual to find a number of very skeptical people around, who question, the time, budget and effort which must go into a UX design for an innovative new product or line of products, when there are easy, low risk alternatives for enhancing existing products.

So how do you overcome this skepticism? How do you sell early UX design to a skeptical audience? Working at Nokia’s Americas HQ [in 2000] I discovered that we can win people over by using usability testing to give influencers and decision makers experiential immersion with a working prototype.

In human factors and usability engineering, employees of the firm are considered “invalid” participants in tests because they might have foresight into how the product operates. The idea is that employs spoil the scientific nature of the test results. [In 2000 human factors and usability testing was very much a science conducted by professionally trained people, in 2016 that is much less so.] However, by carefully selecting a set of “invalid” test participants, you can sow the seeds for future success with the product and gain buy-in to fund the project to completion.

This strategy is not without its risks. If your design isn’t good the project may get killed a lot earlier. Visibility can be a double-edged sword. This article seeks to advise you how to select the candidate evangelists and how to manage the risks of negative reactions by ramping the introduction of participants in product testing as you refine the level of fidelity and usability. The goal is to gain an influential band of company evangelists for your UX design. These people should become the ones who will go forth and spread the word enabling you to get the budget and schedule you need to create a production product.

Creating WAP Applications for Nokia

At Nokia’s Americas HQ in Las Colinas, TX, a younger business school graduate who held a director level position in the business development and product strategy unit, was tasked with proving the value and providing a demonstration platform for the new wireless data technology known as WAP. He hired the consulting firm I worked with to develop this with them. We put together a small cross-functional team of 4 people and we deployed to cubicles on the same floor of the building in Las Colinas, TX. We were surrounded by people who did sales, marketing and strategic planning for Nokia mobile phone sales all across Latin America. For many people on the floor, Spanish was their native language. Our team consisted of: me, officially as business analyst (but later UX designer); Carly, a technical writer; Scott, a we developer; and Terry, a usability engineer.

The goal was to develop a suite of location-based services and to include a transactional or billing component. The purpose was to demonstrate to wireless network operators that money could be made from wireless data applications and that consumers would find the services valuable.

Exploring the Market and Product Opportunity

In the beginning we sat down with 10-12 product managers, strategic planning and marketing people. We didn’t make much headway using our firm’s playbook which was based on requirements capture techniques described by people such as Gauss, Weinberg and Gilb. The problem was that the field was too nascent and the clients had little concept of what they wanted save for the high level statement about location-based services and billing. The breakthrough came when I introduced them to the technique of writing personas. We developed a persona for each market segment. One we focused on a lot in the early days of the project was the “soccer mom” market segment.
It was on this project that I developed my Lifestyle Snapshots technique which was later to be featured in Tamara Adlin and John Pruitt’s book, The Essential Persona Lifecycle. The scenario I wrote up for the book was different from those that I described on my own blog in April 2000 [to be reposted at this site soon].

Once we had a set of lifestyle snapshots for each persona, we began to look at opportunities for the technology to intervene in the lives of the personas – to do things for them that we thought would be valuable. We prioritized this list of opportunities and developed usage scenarios for them.

This whole process took less than 2 weeks as we bootstrapped the project. It was easier to block out chunks of time to pick the brains of the strategic planning, marketing and product management people and gain some consensus and agreement on the specifics – an outline of the product – which personas, which opportunities for intervention with technology, which usage scenarios.
Then the project moved into a new phase.

Rapidly Developing and Validating a Product Prototype

In phase two, we switched to a mode where Terry and I would work together to design the screens for a usage scenario. While I was working on the designs, Terry would devise how to test them. We would collaborate for perhaps 30 to 60 minutes, work alone for maybe 90 minutes then reconvene to compare our work and refine it with small iterative changes. By lunch time each day, we had a testable design, and a set of tests. Scott had been asked to work a “late shift” and he didn’t start until after lunch. We handed off the design to him and asked him to code it up. When we returned the next morning the code was working. It took a few days to reach a critical mass and get a little bit of a pipeline going. At that point we started to schedule usability tests for several hours each morning. Over lunch Terry and I would analyze the results and iterate on the designs. This could take until 4pm in the afternoon in some cases. We’d hand off modified designs to Scott and collect the working code the next morning. We iterated a version of the product every day and tested it with a fresh set of users, every day. All of the participants in the tests were Nokia employees or their relatives and they all signed NDAs. Throughout the process, Carly kept pace with us with user documentation, a formal specification (we were an outsource consulting firm after all), and content for the screen/applications.

To give you an idea of the scenarios we were capable of delivering: “my kids are hungry. I just picked them up from soccer. Give me directions to the closest McDonald’s with a jungle gym facility.” This scenario was well within our range and capability to deliver – although the data and directions were mocked up for our prototype, the business development people had sources for such data even in 2000.

Test as early as possible

We didn’t have the word “Agile” in 2000. However, the process we were developing was partly inspired by eXtreme Programming. Where I had used a FDD-inspired UX process on other line-of-business apps, for this exploratory (“startup”) environment, I needed something more iterative rather than incremental. We got that from the cross-functional team and the daily work cycle of the design-code-test.

Terry and I realized we wanted to test as soon as possible – just days into the 2nd phase of the project and every day from then on. We wanted to be seen to provide frequent, tangible deliverables of real business value [yes, I used that phrase in 2000.] The sponsor was spending budget on this exploratory work and we wanted to show him something quickly. We wanted to build trust with results.
Guideline: Test as early as possible and never later than 3 months into a very large project.

More than an MVP – Enough Functionality to Create Evangelists

The size and scope of the project will determine exactly what you are testing within a given time period. For a big project, you are testing prototypes, possibly in a mockup environment. For smaller projects you may be able to test the real product in the production environment. With large projects, you are testing a prototype. It may be a high fidelity prototype or a low fidelity prototype – in 1998 I was using cardboard cut-out mockups at a bank in Singapore but not with any formal usability testing to provide quantitative feedback. The prototype is likely to feature only a limited set of functionality. Ideally this would represent sufficient functionality you could launch it for a single market segment. [In 2016, the trendy term for this is MVP – minimum viable product – and often it is only sufficient to give insight into the market, not sufficiently complete to launch to a segment.]

If our goal is to turn the testers into evangelists then it is essential that the prototype you do test, has been developed from a thorough user centered design perspective and has some functional integrity. For example, in a home banking web application, it should allow you to perform functions such as account balance check, inter-account funds transfer, and bill payment, completely end-to-end. It must look to the user like a system which has real potential and true business value. The prototype must deliver on 1 or more goals for each persona we’re testing. Achieving goals is a means of delivery true vaue. Goal achievement is something tangible for the user. It turns them into believers. We aren’t just validating small aspects of functionality, we are creating political capital.

Why Are We Wasting Money On Something Fake?

Some of you may meet resistance to the notion of a prototype which by implication is a ‘throw away’. It is worthwhile remembering the wisdom of Fred Brooks’, “Plan to throw one away because you will anyway!” This argument stems from the fact that you can throw away an early prototype and get the design correct or you can wait and take a high risk that you will have to throw away the production system because you didn’t get the design correct. Writing in the early 1970s Brooks was advising us to validate our product designs early or waste lots of time and money reworking a production system later.

When uncertainty is high, options on future products have significant value. Hence, you ought to be willing to spend more to purchase the option to deliver the right thing, to the right market at the right time. The cost of the prototype is the cost of purchasing the option. It is real option theory in action.

Test in at least 3 phases

So you have a system or prototype ready for usability testing. You have met the criteria to be frequent, tangible and of true business value. Now it is time to test. It is time to consider how to minimize the risk of showing an early product to skeptical influencers and decision makers at your firm.

You are trying to achieve a political win for design and design processes. You risk showing a poor design and a badly written piece of software to a skeptical audience. You must consider that bugs in the code will reflect badly on the design and the whole principle of design. To manage this risk, we took a 3 phase approach.

Guideline: You must deliver a complete design for at least one user goal before testing.

Test with “friendly” users first

We started with the clients – the marketing, product management and individual contributors from strategic planning at Nokia.
Initially select a few members from your own team or closely related people. For example members of your QA team; technical writers; analysts; people who supplied requirements; marketing people; sales engineers; anyone as long as they are closely related to your project or product and have some skin in the game. They feel that they have a personal input on what has been produced so far.
You need at least 5 and ideally 8 of these friendly test participants. If you are to get 8 sets of tests results in half a day, you need small test scenarios that you can complete in 30 minutes, or you need multiple test labs and staff to run them in parallel.

Run the usability test in the normal manner. Typically we found 3 or 4 really stupid design errors, or poor choices which needed fixed. We delivered those to Scott when the testing session ended. With a formal usability lab with a 1-way mirror, it is possible to prepare the top priority results from the test while the tests are happening. You don’t need to wait for formal analysis of the results post hoc.
Phase 1 doesn’t have a time period. Instead it has exit criteria. The primary exit criteria is that you have a stable working prototype without functional defects and with all the obvious face palm moment design flaws removed. When you have this: End of phase 1.

Test with “real” users next

Now it is time to run the proper user testing. You will have selected a number of users, perhaps several sets, based on target market, demographics, known user groups, professions, job specifications etc.. Bring these people in and run the same tests on your new design.
At Nokia, we did this with family members of staff from the business development business unit and had them sign NDAs. While family members, they did meet the market segment criteria to represent the personas we were testing.

We wanted to refine our design every day. We felt that 5 to 8 data points was enough. [At the time, this flew in the face of established usability engineering as a science where much larger sample sets of test results were expected to draw reasonable conclusions.] In other words, each time you have enough data, make an improvement to your product, iterate the design quickly. Naturally, you will have to prioritize the changes, some major design ideas may need to be dropped. You will also need to select your developers carefully. Not every developer likes to make rapid changes like this. Select an individual who has “hacker” like mentality and just loves fast lifecycle iterations. [Note: these 2 sentences were written in 2000 before we had Agile and when this style of working was less well accepted in the software development community.]

A worthwhile tip is that you might leave 1 day free in your test schedule for every 5 to 8 test participants. This will give your programming team an extra day to make and test any changes that you ask from them. In other words, develop a bit of a pipeline. Every other day, have a different area of the functionality to test, to extend the lead time for bigger changes. [Note: this idea of pipelining changes and extended the lead time beyond the test cycle cadence is a natural fit for Kanban. We didn’t the notion of kanban systems for product development in 2000.]

Guideline: Iterate quickly, make several well advised design improvements during testing with real users.
Guideline: Select a developer(s) who are suited to the nature of prototyping and rapid design changes.
The result of this phase should be some solid usability test results and a much improved design which has been shown through the testing to deliver on the establish user goals for the design. End of phase 2.

 Phase 3: Selecting the Evangelists

The third phase is where we re-run the tests but this time using company employees who are not directly involved in the project or with the product. They have no skin in the game. Our goal is to turn these people into evangelists. We will do this by demonstrating the true business value of the design and by emphasizing how you got to such an elegant design. As we gain in confidence with our results, we will seek to invite higher and higher level managers to test the product design prototype.

Evangelists must have influence

As an initial strategy for selecting evangelists, you may like to consider the question, “Who do we need to influence?” It might be the third line manager, your bosses boss, or maybe the VP of Finance or the Director of Sales, or maybe the technical support team who are tired of supporting terrible earlier products.

So the first approach might simply be to invite them to the tests. Have a senior official send them an email or a letter, saying that they have been selected to participate in the usability testing of the next generation product and that this is their opportunity to get an early “heads up” on what is coming through the development pipe. That usually works.

If these people don’t fit your target demographic then consider inviting their kids, or parents, or spouses, or golf partners, or whoever will match your demographic but is closely related to the people you need to influence.

Using the Law of the Few

There is a second, more scientific approach, to selecting the evangelists. We can use The Law of the Few described in “The Tipping Point” by Malcolm Gladwell. Gladwell describes three key personality types which we can use to communicate our message. These are Connectors, Mavens and Salesmen. Just a few of these people ought to be enough to tip the skeptics and see that the design message is evangelized throughout the organization. Connectors, Mavens and Salesmen are the people with influence in a community. These are exactly the type of people to whom we need to sell our new design and its benefits.

Connectors are the type of people who know everyone. They are the people you go to, to get the latest gossip in the office. The people at cocktail parties who introduce you to someone that “you really should meet”. Everyone knows one or two connectors because the connector makes it his/her business to know you.

Mavens are people who know lots of stuff rather than lots of people. A company maven may well be a product manager. Someone who is employed to know all about all the competitors and their products. Or it might be someone in a development role, who knows lots about technology or lots about the computer network. The kind of guy that you call when your computer is broken. These kinds of people get around the company and they get to know lots of people. The network guy fixes your computer but he also fixes the CEO’s computer!
Salesmen are the proverbial bullshit merchants who just don’t take “No” for an answer. They could “sell sand to Arabs” and “ice to Eskimos.” They will tend to latch on to one single thing which they see as the advantage and they will go out and sell that advantage. Finding these people is easy. They probably sold you something recently like a lottery ticket for a charity or a share of a syndicated race horse. Influencing them isn’t hard either. Just make sure that they can see that one key advantage and let them go after it.

Usability Testing with Evangelists

So now you have selected the final set of test participants. You have arranged for a senior figure to invite them to the tests. You have refined the tests you will be asking them to complete and the design that you will be showing them. You are ready for phase 3.
It is important that potential evangelists are tested with goal directed questions. Ask them to solve problems of tangible business value. Something which they can see offers value to the user and either profit potential or improved service, to the business.
Run the test much like any other test. Offer some play time at the beginning, go through suitable introductions and make the test participant feel at ease. Present the test questions as normal.

If you care about usability engineering as a science then you will want to keep the test results separate from the phase 2 test results. These phase 3 testers are potentially invalid users and spoil you scientific data from phase 2. However, any negative results obtained from this third group should still be considered valid and you should still act to fix problems uncovered by the evangelist group.
After the questions are complete, ask the participant to provide feedback. Let them talk, let them say what they think. This is your chance to turn them into an evangelist.

It is key that you take the opportunity to sell the user centered design process which led to your product design. Hopefully they will give you an invitation by complimenting the design, or saying something like, “this is much better than KML Corp’s competitive offering”. It is key that you sell the science of the usability testing. The test participant must not leave with the notion that they just participated in a marketing focus group. Emphasize that the design team derive important data from formal testing and that successive rounds of earlier testing have provided numerous improvements already.

If you have done a good job, then your participants ought to leave the test with a warm feeling. They realize that they achieved a number of goals with the new product and that those goals were achieved as easily as might be expected given the constraints of the technology [which with WAP and 2G candy bar phones, were considerable.] Hopefully, they may consider that the design is superior to previous products and better than the competitors’ products. If this has happened then they will go forth among their colleagues and they will tell them, “the new product – I saw it – excellent. Can’t wait to see the sales figures”.

With a message like that circulating, you will have no problems in the next budgetary round when you need to ask for renewed funding for Interaction Design and Usability Engineering.

Epilogue

So what happened at Nokia? Given what has happened at Nokia in the intervening 15 years, this may come as no surprise.
So we had a working prototype – a set of WAP applications with stubbed out back end functionality but nonetheless working. To convert this prototype into a product system was going to take some time and some money. Perhaps a department of 15 to 20 people for some period of months. With on-going support and subsequent versions, we were looking at a few million dollars per year over 2 to 3 years and hence a total budget of $10-20 million dollars. [High tech workers in Dallas are not cheap and in the tech bubble of 2000 were hard to come by.]
Permission was needed. It was a big portfolio level decision. Senior managers were flown in from Finland. They were shown the prototype and given the full pitch on market segmentation, go-to-market strategy and so forth. It became clear during the meeting that they had not seen anything like at HQ in Finland. No one had been able to demonstrate real consumer value in WAP 2G data. Doors were closed. When they reopened the decision had been made not to fund the project. We the consultants went home – job done. [In fact, I left to join Sprint PCS, to design wireless data applications and take an influential role in their 3G rollout in 2002.] The business development people dusted themselves off and started to work on another idea. In real option theory terms, Nokia had chosen not to convert this option instead to discard it. Life goes on.

[The original of this article appeared on June 7, 2000 at http://www.uidesign.net/papers/2000/evangelize.html and can be found via Internet archival services]