In my report I hope to draw conclusions from the application to decide if the product is truly usable. I hope to evaluate my application in such away that will be helpful to me for future projects of this type. I also aim to demonstrate an understanding of the evaluation framework and the importance of testing.
Evaluation Framework and Testing Strategy
In order to evaluate something you have to know what you wanted to create and what the objectives of the creation were, it is crucial to see if what has been created is useful or not. It is also important to critically evaluate multimedia applications such as mine as not only do we have to make sure the application works to requirements we have to take in to consideration the feeling the user gets from using the product, testing these factors is vitally important, we do this by using the evaluation framework. We evaluate also to show that we understand what we have created and show an understanding of how to make it better.
I created a band promotion CD for a band called Verbal Warning, my intention was to enable people using the CD to see who the members of the band are, what type of music they play and to see them in action to give the user an image of the band in their mind. My application may also persuade or dissuade the user from seeing the band in the flesh. In order to evaluate my product efficiently I had to find 5 co-operators which will use the application under my observation. These co-operators should have come from the initial target audience, however, I was only able to find 3 people from my target audience, although I did get two other people of a younger age. I set out to test in both real and simulated environments, this means in the sense of something being real whether the product is usable, does the co-operator understand where they should go next, or where they will go to if they click a particular button. The testing of the simulated environment is the physical button clicks, do they work? Does the music play when it is supposed to?
After implementing the CD is was important to test the product. The testing phase should include the testing of both technical and non-technical functions. The testing of the technical functions are things such as whether buttons when clicked, actually work, for example if a button is meant to be clicked to play a video, does the video actually play. The non-technical functions include whether the product is easy to use, is it clear where clicking a button will take the user (navigation), is the cursor clearly visible all the time? Ultimately the non-technical testing phase is testing the usability of the product. Before testing begun I set out in table format what I would be testing exactly, both functional and non-functional aspects of my application.
The phase of testing uses the evaluation framework whereby I firstly decided what I was going to test, I wanted the user to be sure of where they were going on each button click, this tests both technical and non technical functions in that the button does actually work and that the button click does in fact take them to the page specified. I also wanted to test that the music plays when the user clicks the song title which is labelled on a button. The user had a choice of two videos also which they could play, the buttons for these were labelled video 1 and video 2, when the user clicked on these I videos should play, I would also want to test the stop buttons which would stop the particular video from playing. On the non-functional side I wanted to test that all the pages are correctly titled. Also, it is important to consider HCI the user should b able to see the text on top of the background and the positioning of buttons should follow the same pattern, as do the pictures.
After choosing what I want to test I then had to decide how I wanted to test it, there were several options open to me, these included questionnaires where the user would answer questions on paper about the CD after the using the product. I could have also asked the user to write down their opinions while using it, or I could use semi structured interviews where I only have a few questions that I want to ask, although I may wish to ask other unprepared questions depending on the users answers to other questions. I thought it was also important to test my product on someone from my original target audience which was people of the older generation, similar to the band members’ ages.
It is important to plan for testing, I planned by using a testing strategy which I feel is most effective. I tested my product by using a questionnaire, I firstly gave my co-operators a copy of the questionnaire before they begun using my product, I wanted the user to fill in the answers as they went along. The questionnaire was made up of questions which the user just had to tick the box they felt most appropriate for each question. In addition to tick boxes I also left room for my co-operators to add any other comments. The answers to the questions would help me complete the test plan I had created to record the results of the tests of some of the functions of my product.
Conclusions from Test Plan results
I conducted my test plan according to both the answers to my five co-operators questionnaires and when I used the application on my own. I was happy that all the buttons worked on my main menu page, especially the exit button as it always has to be possible for the user to exit the application, I also thought it very important for all buttons to work on my main menu as any new user would become disillusioned with the application early on if the buttons were not functioning how they should. The main menu is arguably the most important page of the application because this is what the user sees first. Any early faults in the product could result in the user not continuing with it. My user however, was critical of the size of my buttons and felt that there was too much free space on the main menu but all of my five co-operators were happy that the buttons stood out enough to see them. The size of buttons on this page are consistent, however I can see why they have been deemed too small. In order to improve my product if I were to do it again, I would simply make the buttons bigger this would also result in the buttons looking in better positions. Looking back I should have taken more consideration for people with visual difficulties.
During my questionnaire I also asked the users to navigate around the application following my instructions, I asked the user to go through the ‘Meet the band’ section on their own, all co-operators managed this successfully. Four out of five people thought that there was enough information on each page, one though wanted to read more about the band itself rather than each individual. At the time of creation though, I was worried about cluttering the page with a lot of text. I did feel that this user had missed the point a little bit each page was dedicated to one person, thus being each member of the band. I do now realise though that I should have created a page about the bands history. One user did find one of the band members’ names difficult to read, this was due to the font, the band member name is Ian, this user thought it was Jan. I was surprised that only one user picked up on this. I chose this font in the first place in order to be unique, next time I should test that all letters are clear before choosing a font.
The user should also have been able to see the band in action on the videos page. In order to watch a video the user simply had to click one of the two play video buttons. I was disappointed to see that neither of the videos worked visually, only the sound played. The recording of the videos I had intended to use were not videoed by myself, I had a DVD sent to me which lasted 44 minutes long. The initial idea was to capture four 4 minute clips from the DVD using windows movie maker, due to the format of the video on the DVD I was unable to do this. To avoid this problem completely I would next time have to borrow a video camera and record it myself. There was a notion from 2 users that the video page wasn’t welcoming, and their sentiments are shared by myself, I had concentrated my time on trying to get the video functions working at the time of creation that I neglected making the page look user friendly, at the time of having problems I just wanted to get the videos on the page working. I should have made sure the page was consistent with my other pages I would have placed the usual Verbal Warning logo in the top right hand corner of the page. This is present on all the other pages so the user has every right to question why it does not appear on the video page.
The user should also have been able to listen to other music, I had a collection of four songs which I put on to my music page, when the users clicked on the button with the song title they desired to listen to the music should have played, it didn’t. I think the reason for this was that when I uploaded the music to the application and linked it to the button, the music was on my USB memory drive, this means if the memory drive is not inserted in to the computer, the music will not play. Next time to avoid this I would have to make sure that the music is on the computer so that the path name to the file is correct.
I also tested that the return to the main menu button on every page was working, it is very important that the user can return to the start of the application in order to navigate sufficiently, I tried to instruct each co-operator to use the main menu buttons as much as possible on the questionnaire. I felt it necessary to test all of these buttons, and I was happy that they did the job they were meant to. Not only does this test the functional ability of the application, but also the non-functional part, where by the usability is tested, if this button did not work on any page I would not say my application was usable at all.
I also attempted to use a change in cursor to help the user understand the functionality of the stop music button further. The cursor should change from the default cursor to a small speaker icon with a cross through it to show that the sound would stop. When the user hovered over the button in question this function did not work. The usability of the application is not affected greatly as there is still text on the button indicating what will happen if this button is clicked.
Before asking the user to go on to the credits page I instructed them to go on to the Upcoming Gigs page, the users were happy than the information was clearly visible but only one person felt there was enough information there, the other 4 thought I should give some more information about the gig venues and where they are. To meet these needs I should have created another page which showed pictures of the venues and more information about them, I could also have included a contact phone number.
During the test also not only did I watch the application in use, but I also looked for differences in body language. I observed whether the user was confused at any point or whether they were straining to see any text, this is where I noticed one user looking at one of the band members’ pages and they asked me what it said after they had finished on my application.
I do not think that major changes are needed to the design stage of my project, however I would make the buttons the same size throughout the application. In addition to this I don’t think I would use the same font for the band members names, although only one user was confused by one piece of text, I would want to eradicate this completely and just use a more user friendly font for titles on pages.
In conclusion I was disappointed that only one person would go and watch the band on the evidence that I provided, this suggests that my application was barely usable and overall I was disappointed with my creation. The questionnaire results were very similar and this was because there were technical problems with the product, had these problems have been due to poor usability I think I would have got more varied results.