How can a CTO help me test my product?

Do you know where to begin testing your product?

Usability evaluation of software products and web applications is a growing trend in venture and is proving to be an important need when one is concerned with saving money. It ensures that the product is easy to use and that users have a positive experience. When one is steering a lean start-up through new waters, a usability evaluation can often be an invaluable tool that will saves both time and money.

The most basic method for evaluating products is heuristic evaluation. This involves not only an expert review of the product, but also includes a systematic inspection of the user interface needed to be developed. The goal is to find which usability problems in a product’s functionality are most problematic, as well as identifying any challenges to implementing the minimal viable product’s design.  Heuristic evaluation is therefore the quickest and easiest type of evaluation to perform on your product, because it can be carried out at any time – during each and every development phase of the product.

There are several other evaluation techniques as well. For instance, there is ‘usability lab testing’, ‘log analysis testing’ and more. Some of these modes require special equipment and a space to perform them (laboratory tests), while others can be performed in your organization’s normal workspace, without any additional special tools.

Once you consider the type of information you want to get from the test, which, in turn, determines the time required to analyze and reach conclusions, the quicker the test can be analyzed, and consequently, the quicker you can incorporate these conclusions into your R&D process.

An especially skilled Chief Technology Officer can help you incorporate these tests into the development processes of the product, as well as define which tests can be conducted at what stage. Therefore, it is very important to know which is the right test to use to handle a specific problem. In other words, to understand the way these problems occur within a process or flow, you had better choose the right usability lab test rather than using merely a user-based interview.

The following table sums up the different types of tests and method which can be used:

Type of testShort descriptionAdvantagesDisadvantages
Heuristic EvaluationA simple and basic way to test product usability and functionality. The simplicity of heuristic evaluation is beneficial at the early stages of design. Heuristic evaluation requires only one expert, reducing the complexity and expended time for evaluation. Most heuristic evaluations can be accomplished in a matter of days.Quick and cheap; Does not require special equipment; Flexible – no need to keep to the predefined protocol or set a location;Needs a good and experienced Expert to conduct the test. Usually Heuristic Evaluation can cover 70-80% of usability issues and to do a complete overview other methods are required
Think aloud protocolA small number of testers (e.g. 5) perform a number of pre-defined actions and while doing so, voice their thoughts and decision processes. The project team sits beside them and writes down their conclusions.The testers may be either company employees or outsiders selected according to the required criteria. It is easier and cheaper to use the former, but they may not match the required user profiles.Quick and cheap; Does not require special equipment; Flexible – no need to keep to the predefined protocol:The testers are not always comfortable talking aloud while using the application.
User interviewsA small number (10 – 20) users are interviewed by phone in order to get feedback regarding different aspects of the user experience and any problems or difficulties with it.Quick and cheap; well-defined and accessible user segmentsMay fail to identify the problems of less experienced users;Will identify only major problems which users remember, not everything that detracts from the user experience during actual use.This test must be conducted  on already working applications and will be hard to conduct on prototypes
User surveysA representative sample (e.g. of several dozen users) is requested to perform a number of defined operations and fill in a questionnaire about the process. The users report on their own, without the presence of anyone from the teamAllows a relatively large user sample; different user populations can be sampled, allowing the identification of unusual or esoteric problems.Subjective report, not from a trained tester;Users who did not manage to perform the requested task may not report back at all.This test must be conducted  on already working applications and will be hard to conduct on prototypes
Cognitive walk-throughGroups each consisting of a number of users (e.g. 5) plus some of the project team perform an orderly “walk through” of different components of the product and point out any problems or difficulties with each component.This is a group process, with all members of the group allowed enough time to check out the product and perform the requested tasks.Simple and easy; provides meaningful feedback plus new ideas for improving the product.Can be performed as an addition to the Think-aloud Protocol.Can also work on a prototype and not necessarily a working applicationSuccess depends on the intra-group dynamics; excessive dominance by the project team members, or one/some of them, may prevent one or more users from providing their feedback.
Mouse-tracking testsTools such as provided by which collect and analyze a website user’s mouse movements and identify problem areas.Identifies problematic or confusing areas;Relatively few (e.g. 1000) users are needed; can be turned on and off.May slow down website response time; must send information to the external website of the testing company.This test must be conducted  on already working applications and will be hard to conduct on prototypes
Optimization tests (A/B Multivariate)Tools such as Google Website OptimizerAllows improvement of the user by changing different components  within the application.Requires considerable proficiency in carrying out the test; Several alternatives must be prepared.A large sample base is needed;This test must be conducted  on already working applications and will be hard to conduct on prototypes
Usability LabThe user is testing the application alongside a facilitator who gives the user tasks to perform. Behind a one-way mirror, a number of observers watch the interaction, make notes, and ensure the activity is recorded.  Post testing video is analyzed and a report which contains main finding is generatedCan help understand problems in complex processes and application flow; Can be conducted  on a prototype and not necessarily a working applicationExpensive; Can work with a small number of users;  Hard to bring users to the lab; Time consuming; Need professional equipment and space;
Virtual Usability LabUsability Lab testing which is conducted with Virtual Participants.  Using Recorded Video and Audio. (Example to conduct; Not expensive; Easy to define a profile and find the people to participate in the test; Can be conducted  on a prototype and not necessarily a working applicationAnalyzing findings can be time consuming; You don’t meet the users face to face therefor some information is lost;
Automatic log analysisLog analysis is the process of analyzing data about online user behavior which is produced by webserver or application — e.g. date/time of access, tasks performed, and any errors encountered.Efficient in understanding repeated problems in an application. Conducted with real usersNeed large sample of users to support the findings; Need sometime support from software developer to query data;  This test must be conducted  on already working applications and will be hard to conduct on prototypes
Eye trackingEye tracking is the process of measuring either the point of gaze (where one is looking) or the motion of an eye relative to the head. Examples – cased in which disturbing design elements are suspected in changing the user expected behavior.Efficient in understanding very specific problem in selected pages.Expensive to conduct, need domain specific knowledge in running the test; Need unique equipment.
This test must be conducted on already working applications and will be hard to conduct on prototypes
Be Sociable, Share!

About David Rashty

David Rashty, an entrepreneur and one of the early web pioneers, has over twenty years’ experience as a CTO and a CEO. He has been involved in several start-ups and established companies and was the founder of two successful ventures.David is currently using his proven leadership and management skills to act as a Part Time CTO or "JumpStart" CTO for several early-stage ventures; this includes helping them design and develop their product and IT infrastructure.David holds a BS in computer science and an M.Sc in educational technology. He has been a adjunct university professor, given numerous workshops, written several books and articles on information technology and received numerous innovation awards.

Post Navigation