A Voice App that Helps You Prepare for the US Citizenship Test.
Using VR to Test a Voice Experience
The Opportunity
Quizlet Tutor is a voice first app that allows you to learn new topics while driving, mowing the lawn, or multitasking in general.
In part 1 of this case study, I talked about the inability of a dense audiobook to help a listener digest the information. You get something dense out like 12 Rules for Life or a GMAT prep textbook, you will have trouble retaining the information via the audio format.
There is an opportunity for learning companies to use VUI’s to help their customers prepare for important tests and exams.
Think about it, sites like examPAL, Duolingo, Khan Academy and Quizlet permanently remove the need for preparatory books. Making me think that the new medium for active learning is no longer the textbook, if that is the case, then the audiobook isn’t the medium to learn dense material. Quizlet Tutor is what Khan Academy did for e-learning.
My Role: Conversational Designer
Tools: Google Docs, Sayspring, VR Headset.
Timeline: Two Weeks.
Deliverables: Scripts and User Flows.
Team: Flying Solo. Just scratching an itch.
What I have learned so far
In part one of this case study, I just quickly evaluated through rapid prototyping and testing is the need for affordances. When someone didn’t know something, they would panic, and say things like “I don’t know” or “uhh”. This would break my prototype because it didn’t have all the answers.
I assume that I threw the user into a new situation where they had no orientation of the subject matter they where being tested on. My initial mistake was this:
I had them test out a flashcard flow when they hadn’t studied the material.
I addressed in my last article that Quizlet has a “learn” feature that walks you through the curriculum. Like Khan Academy, Duolingo, or any modern learning app, it doesn’t punish you for not knowing the material, it uses machine learning to tailor the courses based on what the user has mastered up to that point.
The Approach. Iteration One.
In this case study, I just want to focus on the “Learn” feature of Quizlet Tutor. I already, established in part one the case study what the system persona is and what the user persona is.
System Persona
A great tutor guides and coaches the student. So the persona need’s to communicate patience and afford responses that helps the user feel like they are in a safe place to learn.
If anything this should allow the user to shed in inhibitions because there are no social pressures to be cool or look dumb while you are alone in your car. If done, right, the user should shed their shells and facades to give themselves to the process.
User Persona
An immigrant to the United States who have multiple responsibilities and not enough resources such as time and money to dedicate to studying for their citizenship test.
My process is as follows:
I cover some use cases with the “Learn” VUI, from those use cases, I will write out some test scripts that go over the uses cases, from there I will draft some high level flows to rough out edge cases and the potential paths a person can take. Using Adobe XD’s new voice prototyping feature I will prototype a Carplay interface that can be interacted with use voice. Then I will test.
Rinse and Repeat.
Use Cases for Learn Feature
Because this is as much of curriculum design as product design, my use cases need to be tailored to the design of the curriculum. For the sake of simplicity, I want to design conversations around the US Citizenship test. Using this components doc as a source, I wrote some scripts that need to be designed.
Primary Use Cases
User successfully learns about the branches of government using the learn feature.
User successfully learns 4 amendments of the constitution using the learn feature.
User successfully learns about the revolutionary war using the learn feature.
User successfully learns about Abraham Lincoln using the learn feature.
User successfully learns about where the civil war started using the learn feature.
User successfully learns about Washington DC using the learn feature.
User successfully assesses competency using the diagnostic feature.
Alternate Use Cases
User struggles with revolutionary war topic after initial first exposure in learn feature.
User doesn’t know the answer to one of the review questions on the branches of government in the learn feature.
User isn’t sure how to answer a question in the learn feature.
Scripts and Flow Chart
Two high level deliverables are the script and flow chart, these are the first iterations that I want to test out.
If you have trouble viewing the embedded script, see it here.
Wizard of Oz Testing Plan
Embedded is my testing plan, essentially, to test the multitasking that will happen when using the app. I’m using a pair of VR goggles that play a 360 video of driving. I will have them wear it while talking through the testing the VUI on them.
If you have trouble seeing the embedded testing plan, see it here.
User Testing — VR Style
Test #1
My first test was with my coworker Denis. He is an italian immigrant who is actually preparing for the citizenship test. I ran him through the test and the biggest take aways was that the MP3 lecture didn’t work. It was too long. While he was paying attention to the road in VR he failed to remember any of the details in MP3 lecture.
Luckily, he sees the potential in flashcard feature. It would be something that he would like to use. He actually did this in real life while driving back from San Fransisco. His wife would read off the questions and if he didn’t know the answers, she would tell him the answer mark that he got it wrong and move on. Next time she asked the question she had a mark saying that it is his second go around with the question.
His insight was this:
For me, she wouldn’t tell me what the topic was, she would just ask me the question, if I got it wrong, she would tell me the answer. If I gave the wrong answer, she would give me a trick to better remember it.
I was wrong about the MP3 lecture. Instead, it would better to jump into the questions and allow the user to fail.
If they don’t know, give them the answer.
If they give the wrong answer, give them a hint.
User Test #2
So I rewrote the script to imitate Denis and his wife’s dynamic and tried it again on my wife, who is an immigrant who actually went through the process of studying for the citizenship test.
She didn’t know any of the answers, but she did like how I just taught her as she failed. In reality, the user could always circle back and review the question.
Test #3
This person didn’t know the answers as well. But when she said “I don’t know” and I gave her the answer, she remembered why she missed it. Hopefully next time, she will remember her mistake.
When she gave the wrong answer, and I re-prompted with “would you like a hint?” she was able to make the connections.
Test #4
Jorge is an immigrant who recently got his citizenship. So he is familiar with the process.
I noticed whenever he paused, he responded better to the “do you want a hint?” prompt instead of being given the answer. It was actually pretty frustrating to him if you just give him the answer after 6 seconds.
So I decided to re-prompt with a hint if there is a no input error.
Test #5
The final test I put to the test the full script. She really enjoyed the VR video and was pretty immersed in the content, making it a ample substitute for the distraction of driving.
My hypothesis was correct. I asked her about three state in the original 13 colonies, and she took longer than 6 seconds to respond. By re-prompting with “would you like a hint?” she was able review some of the states without losing any momentum.
This is by far the smoothest test.
The Revised Flow, The Potential to Fit Most Curriculums.
What I Learned
Deeper into the project, I was worried that the scope of developing such a system might be too costly to produce scalable deliverables.
But I learned the value of testing on qualified test candidates. Every one, knows what it was like to prepare for the US citizenship test and have studied the questions. So they look at this product as a serious solution to their problems. So their feedback is very constructive and and based off what they would need to improve the studying workflow.
My coworker Denis thought the app was a great idea, he said he would use it if it where available.
I’m not certain of the difference between Learn and Flashcards in a voice capacity, maybe Learn has the hints and Flashcards just test knowledge with no hints. It can just go through the flashcards and score you at the end.
I would probably take out the spelling feature, after user testing it is apparent that driving and having to critically focus on something like remembering how something is spelled might be too dangerous. Just focusing on Learn by itself might be the best option.
So in future iterations, I should just focus on Learn. I’d probably change the first prompt to something like this:
“Hi! I’m Your Quizlet Tutor! Your available decks include the US Citizenship test and english vocabulary. What do you want to Learn today?”
This way we just lump together two features like flashcards and learn, and just focus on the core functionality of trying, failing and learning in one smooth flow. As you master topics, new topics will take their place until you have mastered the material. Which will result in:
“Congratulations! You are ready to ace this test!”
