This is an old idea, maybe six years old? My former employer wanted to get a patent for it, then they decided not to pursue the patent. This happens from time to time, patents are expensive. But now, the patent application is in the public domain, which means no one can patent the idea. That is pretty cool, because anyone can build on these ideas, for FREE.
During product development, I run into situations where people tell me, “It doesn’t work”. Oh? How so? “I don’t know what I did, but it broke.” It is frustrating, because reproducing the problem usually involves a fair bit of back and forth, plus more time in front of the debugger. After all that time spent in diagnosis, the fix is usually pretty quick and easy.
So I created UserMetrix. It combined analytics with error logging to automatically diagnose these reproduction steps for me. It reduced the amount of time diagnosing software faults, so I could spend more time fixing them.
Each time someone uses software, I imagine the way they click, and do stuff as a pathway. Someone skipping along a road, navigating the interface and executing pieces of software. That is, until they get to their destination or hit a roadblock. Like an exception or a crash.
UserMetrix would capture the pathway they took, as a list of actions taken or buttons clicked. A bit like this:
- New file.
- New spreadsheet column.
- Rename spreadsheet column.
- New spreadsheet cell.
- Save file.
- New spreadsheet cell.
UserMetrix also kept track of all the roadblocks, the content of exceptions. Stuff like “DoUpdateTemporal - time < lastTime”. All these pathways and roadblocks formed the perfect search space for classical pathfinding.
UserMetrix started by pulling out all the pathways leading to the same roadblock. Then working backwards from this roadblock it would use an A* search to work out the common path that ended at the roadblock. It would count up all the features that happened just before the roadblock. The feature that occurred most was expanded, and the number of features that happened before that tallied. This process repeated until the path couldn’t be expanded anymore. Finally the pathway was reversed to form a list of features - the reproduction steps for replicating an exception.
It also worked for other kinds of roadblocks too, not just exceptions or crashes. Like when software is unclear or confusing to the people using it. Kathy Sierra gave an awesome talk at the Business of Software in 2009. At 43 minutes in, Kathy drops this awesome gem:
Software does not let people make ‘that’ face. It does not let them look like ‘that’. You can do it, but your software is ‘not listening’, it doesn’t care. But yet, if you made that face in front of a human who was helping you, they would do something as a result. They would see that face and they would do something. People are trying to figure out ways to do this in software. How to let users be confused. Of course the AI folks are working on this, right? Well you could make software that could recognise the person’s facial expression, and it’s complicated and it is expensive. Turns out there is a much easier way, which is, you could just let people tell you
, because they know when they are making that face. So just add that button, then of course it is up to you to figure out what to do when they click it. But we can let people tell us they are confused.
Why not use a ‘WTF’ button to capture someone’s confusion or frustration? What about those people who use your application once, click around a bit and then leave, never to be seen again? Both these roadblocks could be pushed into UserMetrix to find common pathways that led to confusion about your software. This data could be used to design scenarios for some traditional usability testing - targeted testing on known problem areas in your software.
I got a fair way with the idea, developed a service and had a handful of applications trying it out. My friend Joshua Lamont also made an awesome video promoting the concept. In the end, I made a few mistakes and wasn’t able to turn a neat idea into a sustainable business. It died a slow painful death despite some enthusiastic early adopters. But with the patent application out in the public domain, I can talk about the idea and techniques. So while although the business may have failed, the idea lives on.
Comments:
You can join the conversation on Twitter or Instagram
Become a Patreon to get early and behind-the-scenes access along with email notifications for each new post.
Hi! Subconsciously you already know this, but let's make it obvious. Hopefully this article was helpful. You might also find yourself following a link to Amazon to learn more about parts or equipment. If you end up placing an order, I make a couple of dollarydoos. We aren't talking a rapper lifestyle of supercars and yachts, but it does help pay for the stuff you see here. So to everyone that supports this place - thank you.