At this year’s IWMW 2015 event Neil Allison, the User Experience Manager at the University of Edinburgh will facilitate a half-day master class on “Usability Testing in an Agile Development Process“.
In this guest post Neil summarises the approaches taken at the University of Edinburgh to agile usability testing. Note that this post was originally posted on LinkedIn.
Making Usability Testing Agile
Spaces are still available for the half-day workshop on
“Usability testing in an agile development process”
I’ve been running regular usability testing observation sessions as part of an agile project we’re running at the University of Edinburgh to enhance the new Content Management System we’re delivering, and to bring the development team closer to the end user.
Fitting usability testing into an agile process is quite challenging (we operate 9 day iterations spanning 3 weeks at a time) as time is always tight, but the methodology I’ve put together is working really well.
At the end of 2014, I ran an open invite session for web publishers, developers and project managers from around the University, to give colleagues an insight into how we’re doing this, and to allow them to participate in the process.
In this post (based on a something I wrote for our team blog), I’m basically writing up what I said so that readers can get a good idea of how I fit regular usability testing into a very tight development schedule.
Original post on the University of Edinburgh Website Programme blog
24 staff from around the University joined the CMS development team to watch 3 usability tests and contribute prioritised issues to address.
I’ve included links to the slides and am happy to help anyone who wants to try it for themselves. It’s pretty easy and the resources I use are freely available.
What we’re trying to achieve
The first thing to be clear about is that this isn’t about agile development. It’s about achieving regular, rapid, inclusive usability testing that results in measurable improvements, and with minimal overheads. So it will work for you regardless of any development methodology you’re following.
I’ve run a usability testing training course for years and had over 600 colleagues around the University attend. As Steve Krug says, “It’s not rocket science” and I think most leave my training seeing just how easy it is to get insight into the effectiveness of their website or application. Many go on to put the training into practice.
Usability testing training session overview and participant feedback
But there are challenges, and I overhauled this training a couple of years ago to cover what happens after you’ve done a few tests and identified what you feel you need to do to improve.
The challenges staff have raised with me (and I’ve encountered myself at times):
- Getting the go ahead to use your time on usability testing
- Getting colleagues to take on board what you uncover
- Getting fixes to problems implemented
And challenges such as these aren’t just faced by people like you and me. Usability and user experience professionals the world over encounter blockers such as these every day.
Caroline Jarrett and Steve Krug presented research on the topic: Why usability problems go unfixed
I have additional challenges playing the role of UX Lead for the development of the new University CMS, the main one being that this is not a formally recognised role within Information Services and there are no formal usability-related processes in their approach to software development. But on the plus side, this has given me scope to experiment and innovate and it’s helped drive me to where we are now and the approach that we take.
Our process
I’m going to say right now that there’s nothing particularly innovative going on here, and that I didn’t invent any of it – I’m just standing on the shoulders of giants. Mainly Steve Krug, with a bit of help from David Travis.
The majority of what is covered below (minus a few tweaks) is from Steve’s fantastic book: Rocket Surgery Made Easy. After running sessions for a few months, I also discovered the Gov.UK user research blog which highlighted that they’re just a bit further down the same road I’ve taken us.
Have you had your recommended dose of research? – Gov.UK user research blog post
What we do:
- Get the right people in a room
- Watch a small number of short sessions with users doing something
- Prioritise the issues we see
- Collaboratively consolidate their priority lists
- Agree actions for usability issues
- Repeat every few weeks
Who are the right people? Basically everyone with a stake in the development. No exceptions. Our time is so tight that I’ve negotiated within the team to ensure that at least one representative from each area of activity is present. Ideally the whole team would be present to observe but it’s not an ideal world. So this means I always have at least: a project manager, a developer, a service manager, and a training and support representative. Sometimes I manage to get a more senior stakeholder in the room for at least some of the time too. So a minimum of 4 colleagues see what I see, and sometimes we’ve had 9 or 10.
What do we watch? We watch real CMS users undertaking tasks in usability testing sessions that I facilitate. The focus for the session is agreed a week or two in advance so that I can plan scenarios and make sure we have a representative environment to work in, and also so that the team can focus my attention to whatever they feel is most appropriate. Typically this is an area which is causing concern or an area where we’re about to begin adding new features.
How many participants? In the presentation I use the graph from Jakob Nielsen’s famous article, “Why you only need to test with 5 users” but what I actually said was “As many as you can fit into the time you have (so probably not very many)”. In practice for us, with 3 hours allotted for this activity, we watch 3 participants for about 20-30 minutes each which leaves us with enough time to discuss at the end.
Jakob Nielsen: “Why you only need to test with 5 users”
How do we prioritise? We all make our own notes, and at the end of each participant’s session, we each fill in a form independently that logs the top 3 issues we observed. So at the end of the session we have each filled in a form with 9 blank spaces. We may have written down the same 3 issues for all 3 participants, but not usually.
The CMS development team discuss the issues they’ve noted during the usability testing session.
How do we consolidate? In the early months we just did this through an open discussion, but I found it quite hard to keep the discussion on track and therefore on time. Time ran on and people needed to leave so getting real consensus was difficult. And then I remembered David Travis’ usability issue prioritisation flowchart and more recently we’ve been using this. This has helped keep the post-test conversation to about 30 minutes and provided greater transparency about how we prioritise.
And so out of this, we have a list of prioritised issues that we assign to members of the team to action. The action might be:
- Get this prioritised for upcoming development (because the solution is “obvious”).
- Make changes to our training and support processes.
- Add to challenges for future prototyping of new interfaces and processes for additional testing (because we don’t have consensus on how to improve the situation, or the best solution would be costly to implement so we want to be assured it’s right before we commit the development time).
Benefits of this approach
For the development team:
- We get closer to our CMS users – and immediately see the impact of our efforts
- We gain shared insight & experience
- We confirm ownership of the priority issues
- What to fix immediately
- What to do better next time we’re developing in that part of the system
- What we thought was a problem that turns out to be something we can live with
For me:
- The process keeps set up and organisation of session to a minimum
- No report writing – just a single wiki page logging what we did and a table of priority issues and actions
- Doing this regularly moves the culture of the team on, emphasising CMS usability on the development agenda
What we need to do better
I have two challenges that I continue to work on:
- How do we minimise usability issues making it in to the system in the first place?
- How do I get more of the right people in the room, more often and for longer?
How do we minimise usability issues making it in to the system in the first place?
This is tricky because we’re working in Drupal, an open source CMS. This means our developers rarely create stuff from scratch. They’re drawing on a community of developers’ existing work which means the cheapest solution is to just take it as it is. We have inconsistencies in presentation, labelling and functionality which need to be prioritised to be addressed. This of course gets us back to why I’m doing this testing in the first place.
Developer time is so tight, it’s difficult sometimes to find the space to discuss just how we’d like something to work to the level of detail we’d all like. Ideally I would work with developers to understand what was cost effective to work on and what we should probably leave as is before I went off to prototype and conduct early usability testing. But this can’t always happen and I have to work with what I receive from developers as a first pass. However, going back to why we’re doing this testing, the more our developers see real users interacting with the product, the more likely they are to make better decisions independently (not that our developers don’t make a lot of good decisions of course!) and we get more (more) right first time.
How do I get more of the right people in the room, more often and for longer?
As I mentioned earlier, we have agreed a minimal attendance from the team but the benefit of this process comes from everyone seeing the same thing with their own eyes, and discussing it together. Everyone on the team agrees it’s a very worthwhile initiative but unfortunately we all have other pressures and commitments. We continue to discuss and evolve our wider working practices and I hope that this activity can further enhance the perception of value in usability research on the project.
What are you waiting for? Try it yourself!
So there you have it. Not that hard at all, particularly if you just take on the same materials and processes I have. The benefits are cumulative I think. With every month that you get stakeholders back together to watch users the greater the momentum behind the user focus grows.
We use Steve Krug’s form to log 3 issues for each participant, to be discussed and prioritised at the end of the session.
Have a look at my slides and drop me a line if you have any questions. All the resources and further reading are in the slides, but essentially all you need is:
- Steve Krug’s free resources from Rocket Surgery Made Easy
- David Travis’ usability prioritisation flowchart
You can download my slides from Slideshare if you want to dig a bit deeper.
Sessions slides on Slideshare.com
After the event – the feedback
Colleagues from across the University that came along to our open session were incredibly positive both on the day and after it in comments on the session wiki page.
I think the session worked in 3 different ways:
- Users of our current CMS got a preview of how they’ll undertake key tasks in the future. We were open about where we’re up to, including the flaws we still need to deal with.
- Members of our web management community got to highlight issues they saw in the new system, and contribute to an open and democratic means of prioritising the severity of issues.
- Developers, project managers and website owners gained some experience of a way to approach usability testing that is efficient, inclusive and more likely to result in improvements being made.
A few snippets from the feedback I received:
“…[the session] gave me a few good ideas to use when user testing my own websites, particularly the flowchart for prioritising issues and the instructions for usability test observers… [I] will be trying these out in February when testing a website we’re developing… I also enjoyed collaborating with other university staff.”
“…[the session] highlighted the importance and difficulties of user testing someone ‘live’. I noticed that myself and other participants began focusing on aspects of the design which we thought should be improved regardless of whether those aspects actually caused the participant any issues. So I took away from it the realisation that a bit of focus and discipline in observation is needed…”
“It was good to see users in action and how the new university website is shaping up. It was an interesting insight into user testing and definitely gave me ideas for our own user testing. I think the prioritisation flowchart was really useful and I think I will use this myself in the future. Another thing to mention, is that it was good to see other staff from the university and collaborate.”
“Overall, excellent… The slides …and notes I took will help myself and colleagues greatly as we undertake user experience sessions in the coming weeks… The session was extremely useful and provided valuable insight and guidance on how to run UX sessions that provide measurable results.”
About the Author
Neil Allison is UX Manager at the University of Edinburgh Website Programme. Here he steers the evolution of the University website’s information architecture and the user experience of the corporate content management system. He also oversees the provision of training and support to the University’s web publishing community.
The University of Edinburgh is a large, research-led institution and a member of the Russell Group. The student body totals almost 34,000 with over 11,000 engaged in postgraduate study, supported by over 12,000 staff.
The University Website Programme began life as a project team in 2006, becoming established as a Programme two years later. Its function is to manage the corporate content management system (used by over 1000 staff in around 90 business units), promote and support best practice in website management and to facilitate the ongoing enhancement of the site in areas of cross-institutional collaboration. The primary focus at present is the development of a new CMS (using Drupal) and the migration of websites, users and processes to the new platform. This transition will be completed by the end of 2015.
Contact details
- Email: Neil.Allison@ed.ac.uk
- Twitter: @usabilityed
- Blog: www.usability-ed.blogspot.co.uk
- LinkedIn: neilallison
- Web site: www.ed.ac.uk/website-programme/neil-allison
View Twitter discussion: [Topsy] – [bit.ly]
The post Guest Post: Making Usability Testing Agile appeared first on UK Web Focus.