Session Based Testing, Exploratory Testing and my Questions technique

SB – Session Based Testing - 

Technique Element

Sub section – Approaches

Since Jonathan and James Back (satisfice.com/sbtm) documented their Session Based Test Management approach combining exploratory testing, risk management and ‘management’ oversight there has been a lot written.  Hopefully by now most people know the benefits of exploratory testing and some of the various methods of recording that activity.  In this article I hope to share a brief overview so we are on the same page.  A list of the main benefits and some minor drawbacks.  And finally the question technique I apply when using this in my day to day work.

Overview:

The Session Based Testing approach was developed for a project to allow their test team to ‘organise their work without obstructing the flexibility and serendipity that makes exploratory testing useful’.  They needed a way to keep track of what was happening, so they could in turn report back to a ‘demanding client’, while ensuring the time spent created the biggest return on investment. 

Essentially this is structured

exploratory testing

to help organise thoughts, capture questions and insights and allow rapid feedback.  Key elements to this approach include;

• Each session is chartered (associated with a specific mission)
• Uninterrupted (as much as is possible)
• A template is used to record the details of the mission and findings
• Reviewable ( a ‘report’ is produced to document findings and questions and the tester is ‘debriefed’) 
• Time-boxed (with flexibility but generally no more than 2 hours)

In my opinion, there are a number of flexible points in the approach and tips that are worth being aware of, especially if you’re doing this for the first time;

• I don’t think it matters if you call it a charter, mission or focus.  As long as you generally stick to your subject, although picking one might help when sharing for consistency.
• Interruptions should be avoided if possible.  On occasion I’ve shut down Outlook and put my headphones on for these types of sessions.  At one time I even had a red flag on my desk which indicated do not disturb unless it was urgent. 
• There are templates available or you can create your own like I did.  Again it’s useful for consistency to stick with one you’re happy with.
• Reviewable.  A lot of focus is on ‘management’ reviews but team, peer or even self is fine, as long as what you find generates actionable insights rather than getting filed away never to add value.
• Time-boxed.  If you start small with something very specific that’s a good way to get a feel for this technique and learn to focus.  I can sometimes be like the dogs in ‘Up’ and be distracted by squirrels!  Learn to note where the squirrels are and why you need to look at them later.

Question technique and template:

I admit that I often use this as a mental reminder, rather than something to populate, as my preference is to speak to a developer on my team immediately after a session to investigate or question.  (I don’t raise bugs I describe behaviour and in writing this, that’s probably what my next post will be on.  I’ll add a link to it on here when done.)  Only if this isn’t possible due to availability will I actually fill things in from the notes I have taken during sessions.  For me, this is a disposable document with a short shelf life used to capture, discuss, resolve (or not), and most importantly discard.

I’ve reproduced the template in bullet form rather than embed a PDF or word document, that way I hope it will be easier for you guys to take away and make your own.  When you get to the questions you might find, as I do quite often, you will remove some before you start as not applicable, or you won’t have filled some in when you’ve finished.  It’s supposed to be flexible like that but you should take a moment to understand why they are not populated or applicable to the session as that may prompt some other thoughts.

The template:

• The Basics: Date; App/function under test (brief description); any other useful information depending on your context
• Any dependencies vital to the testing (connections, files, data, hardware etc. this helps make sure you have them before you start)
• Any information that is useful such as material/learning’s from previous sessions, personas to use, environments, tools etc.
• Test strategy (a consideration of techniques you might use as a flexible plan is often more useful than no plan, but don’t be afraid to improvise as that’s half the fun and discovery may make your plan obsolete quite quickly)
• *Metrics (see rant at the end of this post)
• The questions: (with a brief reasoning for them)

o

What do I expect? (even if it is something brand new I always have some expectations)

o

What do I assume? (sets a context that I can query as I go)

o

Are there any risks I should be aware of? (to execution, the system, helps anyone else reading have context)

o

What do I notice? (behaviour; usability)

o

What do I suspect? (things that I feel, not always based on facts but that I don’t want to lose)

o

What am I puzzled by? (behaviour that doesn’t feel right)

o

What am I afraid of? (high priority concerns about the item under test)

o

What do I appreciate/like? (always good to have some positive feedback)

·

   Debrief (originally between the tester and a manager there’s a checklist of questions on

satisfice/sbtm

.  My version is more often a conversation with the developer with questions or queries, but can also be with the product owner or stakeholder depending on what I find/context. I’m not saying don’t do this, rather do it only where it’s going to add value.

This post is getting a bit longer than I’d hoped but I feel it’s important to summarise the benefits and possible drawbacks of using this method so there’s a balanced view.

Pro’s

Con’s

Allows control to be added to an ‘uncontrolled process’

Can be harder to replicate findings as full details are not captured

Makes exploratory testing auditable

As a ‘new’ technique it has to be learnt

Testing can start almost immediately

Recording exploratory testing (rather than brief notes) can break focus / concentration if you're more worried about doing it

Makes exploratory testing measurable through metrics gathered

Time is required to analyse and report on metrics

Flexible process that can be tailored to multiple situations

Time is required to discuss/give feedback to potentially the ‘wrong’ person

Biggest and most important issues often found first

Can help explain what testers do to clients, stakeholders and the uninformed

Given all the above, if you have to justify exploratory testing, (notwithstanding you should be looking for a new job!) then using session based test management either in its original form or some hybrid version could be the convincer you’re looking for.  In the table above, ‘management’ will generally only see the Pro’s column which covers a lot of the things ‘they’ will worry about.  But seriously, look for a new job!

*Metrics: I personally don’t think these are useful for virtually anything (oh, more controversy!), but if you absolutely have to report back to someone, a manager who knows little or nothing about what testing really is for example, here are some metric examples.  How much time you’ve spent such as start/end times on actually executing testing; blocked; recording findings; actionable insights; questions/queries; potential problem; bugs; obstacles; screen shots or some other method of recording or documents to show any issues to help replicate them.  

Using Personas and the Relationships Between PToT Elements

PS Personas – Technique Element  

Sub section – Approaches

There are lots of relationships we have to consider in testing.  In this post, I’ll briefly discuss those relationships and how the Periodic Table of Testing can be used to map them.  Then share a real-life example of how using the personas ‘thought technique’ lead to using other elements on the table.

Any idea, technique or approach can only take you so far without some view of those things surrounding it.  Even a Hermit (a person wanting no contact with others), no matter how isolated, has relationships that need to be considered such as their surrounding environment. 

Understanding relationships can often be instrumental in identifying appropriate scope that helps ensure we deliver quality in our products.  The Periodic Table of Testing is exactly the same.  A Technique Element can have a relationship to a Testing Element and in turn a Testing Element can lead to (have a relationship with) a Technical (or any other) Element.

Real example:

Below is an example of how a Technique Element lead our team to a Testing Element that helped describe our relationship with our Customers.  By creating Customer Tours or Journeys we could then mimic the Customers behaviour, particularly when using Personas to navigate the system in a particular way.  Those Tours and Journeys then lent themselves to a Technical implementation through automated tests and the creation of Living Documentation.

Working on a project to create a customer portal to access mortgage account information was a great opportunity to introduce personas.  I’d read a lot about personas but the main takeaway for me was how they could be used to highlight key differentiators.  For our project, the key differentiation was the accounts status at the point of use.  I’ve read and seen a lot of information on personas and some recommend highly detailed and complicated outlines.  For me a lot of the detail in those were superfluous and didn’t add any real value.  For projects lasting years they could hold some worth but for me were distractions from the main point of them.

Back to the project and our main differentiation.  Mortgage accounts have several status variations including the account being up to date, in arrears, with an arrangement, in litigation, in possession and so on.  We used different personas to represent those different states. 

With input from the team we even used the names of the personas we created to represent variations in surnames to see how they would be displayed in the UI.  And so, Sally Steadman, Adam Thompson-Pritchard and Olivier O’Connell amongst others were ‘born’.  While the personas had genders, ages and key personality traits their development didn’t go much further as the status of their accounts was the key differentiation required.  Once we had the personas and a shared understanding of what each one meant we expanded the idea to other elements.  As well as creating customer journeys for them and noting the different information and help items they would see, we wrote feature files for them that became both our automated testing and in turn our living documentation. 

Thanks to our shared understanding we were able to create a ‘Preview’ version of the site and added fake services.  This meant you could register and sign in as one of the personas and explore or complete user journeys just as the Customer would.  We used these to execute our automated UI tests giving us stable responses.  Cool stuff I thought! 

However we might conduct testing and wherever our starting point; the relationships of different techniques and methods must be considered in our quest to investigate and add value to the best of our abilities. 

References:

Wikipedia: https://en.wikipedia.org/wiki/Persona_(user_experience)

Usability.gov: https://www.usability.gov/how-to-and-tools/methods/personas.html

Alan Cooper Origin of Personas: https://www.cooper.com/journal/2008/05/the_origin_of_personas

Agile introduction: http://www.agilemodeling.com/artifacts/personas.htm

Generic Testing Personas: http://katrinatester.blogspot.co.uk/2015/01/generic-testing-personas.html (great example of minimal personas)

Just for fun!: http://blog.eviltester.com/2011/10/push-your-software-testing-personas-to-the-limit.html

The Periodic Table of Testing, an introduction and history

Firstly, thank you for taking the time to read my blog.  If you have any comments feedback or questions I'm eager to hear them so please get in touch. 

I'll be using the blog to share my thoughts on testing, feedback on events I attend and to share the things I find most useful. 

But the primary reason for the blog is to document my investigations and journey's through the world of testing using my

Periodic Table of Testing

.  

Over time I hope to navigate through the table as I ask myself, do I understand what this is, how it works and how/when to implement this in the projects I work on.  After all, theory and ideas are all well and good, but if you can't then apply them, what good are they? 

The table is an ongoing work in progress and I expect it to change over time.  It could grow, have elements removed or even have a new sections added.  For example, I'm not sure if interpersonal skills should have its own column or section as there's elements in the table but they are so important perhaps I should highlight them? 

The table takes its inspiration from many sources.  I originally created the

Periodic Table of Data

 back in 2011 while working in a Business Intelligence role as a way to see how new data could fit into our existing framework.  It was also a way to understand what we already had. 

The idea was published in the Testing Planet in March 2012 and the article is available on the

Ministry of Testing website

. 

While I've been playing around with this idea for some years there have been a number of recent influences I'd like to highlight in spurring me on to finally publish.  I attended NWEWT in March 2017 which is the

North West Exploratory Workshop on Testing

.  The workshop was on growing testers and I thought this could in some small way help testers navigate the world of testing.  Or even be used as a visual heuristic of considerations for projects.  Another contributor was

Ash Winter's Wheel of Testing

 and how he used it as a tool for the testers he managed.  

There are so many more influences I'd be here all day but a quick mention to Chris Pearson and Andy Lawrence at Computershare for supporting my crazy ideas to do stuff and beginning my education on all things Agile respectively.  

So that feels long enough for an introduction, please leave any comments or thoughts below.  Thank you.