Gameplay test at Roskilde Festival
June 2023
Roskilde Festival
UX Research Intern
How to organise a scope that is too broad for a user research study?
For a whole week, the team and I conducted user tests on over 90 festival attendees inside the barnyard of The Circular Laboratory, which is located in the middle of Roskilde Festival's facility. We used various methods to collect user data.
From left to right: Camilla, Oskar, Andreas and I.
Thrown into new surroundings
My first day of the internship was at the Roskilde Festival, where I was not a part of the planning process for user methods and had not yet completed my onboarding tasks. Heck, I have never even stepped foot inside the Roskilde Festival before. The team behind the planning phase was a former UX researcher named Elisa, whom I had never had the pleasure of meeting, and my then-colleague Danny. But since he lives in the UK, I was the only UX Researcher.
Too broad of a research scope
At the festival, three other team members (Andreas, a sound designer and a historian) and I conducted user research based on “trial and error”. It resulted in a week of gathering data from 92 participants. However, it lacked consistency because we forgot to ask the same questions for each user because of the amount of 7 methods and 46 interview questions. Due to the former UX Researcher who planned the project had to end her internship, the communication about the research goal was not clear and it resulted in many different demands on what to test and which questions to, to the extend of it being too much to keep an overview of.
The methods used were different depending on the day:
​
-
We audio-recorded interviews.
​
-
Had an available online survey.
​
-
Usability tests.
​
-
Observation notes.
​
-
Screen recorded the game plays.
​
-
Tested out a paper prototype.
​
-
Tested out a QR code for extra special card decks.
Me observing a participant
Treating me like a professional
From day one, the other 3 team members asked for my advice and what to do next as if I had been employed for a long time. This made me more confident and made this adaptive situation easier to navigate in with ease. A laid-back and informal work environment that pushes me out into “the wild” and asks me to take charge is where I shine, based on what I experienced during this research project. I led the team by "trial and error" and found effective solutions for participation differentiation sorting and organisation for data collection gathered every day.
Result
The UX department and I transcribed 40 audio interviews, analysed 15 video recordings and wrote a report of 139 pages. While we found many insights to present to the other departments, the data was not as strong due to inconsistent data collection.
Quote
"An example of Helena's capabilities in leadership was of the
Roskilde Festival project. She independently conducted comprehensive user tests, managed data analysis, and delivered impactful presentations to our Game Design team. This work directly influenced our game optimization strategies."
​
- Andreas J. - Founder of GAMUCATEX
Roskilde Festival 2024
June 2024
Roskilde Festival
UX Research Intern
Collecting data yet again at a music festival
With experience from last year, I and the team used this to come back even stronger and learn from our mistakes. This time, I was the lead of the planning and execution part of the research, but not the analyzing phase, because my internship had to end after I graduated with my Master's degree.
Project management & stakeholder collaboration
During a long-term project, multiple departments at GAMUCATEX had different tasks to solve before setting up camp at the Roskilde Festival. I communicated frequently with The Circular Lab, ensuring that information was gathered and that we compiled according to their demands and guidelines. Within the company, I facilitated the research scope and which features of the game to test for the development team. Last year, we tested a paper prototype of an external shop with additional campaigns or DLCs. The shop also contained deck bundles with beautiful illustrations and explanations of how Norsemen utilized and reused their natural materials. This year around, I wanted them implemented into the game and, ultimately, made that a reality by communicating the need for these features early on. Last year, we did not focus on entertaining activities. While the main focus was to gain feedback on our game, for this year, we received much help from the marketing team coming up with fun activities that ensure the festival attendees will remember GAMUCATEX as more than a game company, but engaging and informing them about the Vikings age through fun activities such as drinking from Viking horns.
Qualitative research
​There are countless data collection methods, such as questionnaires, surveys, eye tracking, contextual inquiries, etc., often divided into two categories: qualitative and quantitative. Not only did we choose a qualitative approach since we wanted to test the usability of the newest version of our game, but we also wanted to pick a social interactive method since we did, after all, collect data in a fun environment such as the Roskilde Festival. Using the think-aloud approach paved the way for our participants to express their opinions through how they interpret them through their worldviews. While quantitative surveys can be wise for collecting demographic questions from a large sample, they are not the most socially interactive approach, as we especially learned from last year's data collection at The Circular Lab. The methods we chose are explained below.​
Usability test
Throughout testing, we used the usability test approach, which was meant to disclose information about how the participants interacted with our game. Elizabeth Goodman, co-author of Observing the User Experience (2012), explains that it can help identify issues; for instance, can the participants complete the tasks we designed for them to try to solve? Do they understand the language and use of wording in the game? Before the happy festival attendees tested our game we gave them a disclaimer where we explained that the purpose is to test the game, not them and their gaming abilities. This is because we know from past usability tests that some participants have felt insecure due to not understanding each element of the game. We wanted to make them feel at ease and see the test as fun. This is followed by showing a clearly explained consent form since it is beneficial to explain their rights and how we intend to store their data. We explained here that we wanted to audio- and screen-record the test to improve the game.
Audio- and screen recording
To gather consistent data, we designed an interview guide that asks each participant the same questions. Asking questions can be an ice breaker, as it makes the participants open up more and allows them to express themselves more. Throughout the gameplay, we did not intend to give them directions in terms of where to click; in fact, we would rather ask them to tell us what they expect they have to do throughout the game. The reason for screen recording is to analyze pain points and reading speed. It can also help us analyze the transcripts of audio-recorded interviews. The interview might go by fast, and some details will be lost in the audio files that the screen recordings can pick up. For instance, if a participant talks about a specific card but does not mention its name, we could then see which one was referred to in the screen recording.
Methods we did not include
​Based on what we learned from last year some methods are just not suited for collecting data at a music festival.
1. Participant sorting list
Last year we had a printed-out table where we wrote, for instance, “Participant 7” and the timestamp “13:45”. However, we found that it was difficult to keep track of due to being two researchers with each of their participants. This time around, we planned for each researcher to say into the audio recording microphone, “This is participant T6”, meaning the first alphabetical letter of the researcher's name and the number of participants the researcher has collected data from.​
2. Observation notes
While observation notes have their perks, we found last year that they were difficult to interpret when handwritten and the lack of proper Wi-Fi connections on an iPad. This year, we also focused on the fun activities, as mentioned in the introduction, and therefore, there was not always an assisting researcher behind each laptop to take notes. Any interesting or odd observation could be addressed by asking the participants about their reactions and decision-making. Karen Holtzblatt, a big contributor to the contextual inquiry method, once stated the importance of researchers making sure the users elaborate on their statements, and we took this advice to heart and in a practical setting
3. Headsets
Data collected from last year showed us that most participants enjoyed the noises and background music in the game we tested, and therefore, it was not a research focus for us to explore this time around. Also, we wanted to converse with the participants while they played our game; therefore, we excluded headsets.​
Arriving on the first day of Roskilde Festival 2024 marked a special day for me since it meant that I had been a UX Researcher for GAMUCATEX for a whole year. I have learned so much during, and it reflected in a more successful user research project this year around.
Reflection
What worked well?
1. Interactive playfulness
The main purpose of having a booth at The Circular Lab is to test our product. However, it is also to give the festival attendees a fun experience. It is a give-and-take scenario where we deliver good service by giving the attendees the opportunity to try out a Viking hamlet and, in return, ask them for their feedback on our game.
2. Knowing out limits
​Last year, we tried out various data collection methods on far more participants. While it was fun, it was also confusing keeping track of all the different questionnaires, both qualitative and quantitative, different printed-out artefacts, participants' sorting lists, observation notes, etc. This year, we reduced the number of questions and tasks we gave the participants, but this ensured better organization, data sorting, and more consistent data.
3. Presenting our sustainability values
Last year, we tested an energy-saving button inside a previous version of our game and asked a few participants for their opinions about sustainability in gameplay. This time around, we ensured to ask far more regarding this subject matter while showing newer green features that we had not yet gained any feedback on. Here, I am referring to the green bundles, our shop, and an additional campaign.
What can be improved?
1. Clearly present the data collection plan
Replicability might not be as strong as we wanted due to designing a too-broad research scope. Some features were tested that were not initially planned. The UX department’s goal is to keep the research small in order to gain in-depth data, but it is also about including the wishes of other departments in terms of what they want to gain insights about. Finding a middle ground between the two is a struggle. That is why anyone in UX has been taught how to collaborate with multidisciplinary fields and various stakeholders while embodying a holistic mindset to see how a research plan can accommodate different needs.
2. Kill your darlings
Working on this long-term project taught me to know when to kill my darlings, or in other words, to let go of preconceived assumptions and plans about how I think the plans should be executed and let the practical environment inspire us to conduct data collection differently.
3. Ensure successful audio recordings
At loud facilitation, such as The Circular Lab during the Roskilde Festival, we can advise the researchers to hold their phones closer to themselves and the participants while audio recording. Some audio files were nearly impossible to hear because the phone was placed on the table. It may be uncomfortable for the arm, but it will serve you when transcribing and analyzing the data afterwards. And last, audio recording from a laptop works extremely poorly, so I advise you to use an app on your phone.
Ended as I started
Quote
"I saw an impressive growth in UX methodologies, leading to seniority amongst the other UX'ers"
​
- Andreas J. - Founder of GAMUCATEX