Thursday, July 20, 2017

In Two Minds

Jerry Weinberg's definition of quality is well known. It is generally applied to encapsulate a relationship between a person and a product, at a particular time, and goes like this:
Quality is value to some person
It is intended to be a practical tool, and I think Weinberg would agree with something like this as a gloss for it: theoretical assessments of quality — perceived quality — are less important than those which are motivated by action. For example, a property is worth what someone actually pays for it. Without the action, it's just philosophy. What someone is willing to pay, or sacrifice, determines the quality (to them) at that moment.

I've thought about this definition a lot over the years. In particular I've found myself speculating about the granularity of the definition. Back in 2012 I was wondering whether it was interesting to consider quality in terms of the aggregation of a set of qualitiesmore recently I was thinking about the way that product quality and its effect on quality of life might be interesting; and just now I've been worrying away at the possibility of holding conflicting views of the quality of a product at the same time.

Although there are numerous examples in Weinberg's work of multiple people with differing opinions of a product at once, I haven't found any where a single person has that. Here's one relevant extract, from Quality Software Management volume 1, page 5:
For different persons, the same product will generally have different "quality," as in the case of my niece's word processor. My [complaint about a bug not being fixed] is resolved once I recognize that to [my niece], the people involved were her readers; and to [the word processor developer], the people involved were the majority of his customers.
One of the things I find intriguing is that the definition, and its common usage, together seem to suggest that a person is only able to assert or demonstrate or alter their assessment of the quality of a product at certain times: when they pay for the product, and when they are making use of the product. So I tried some thought experiments.

In the first, I imagined that I might be in a position to buy a Bentley:
  • I believe that the build standards of a Bentley are much higher than cheaper cars. I will pay more for higher build standard. This is a measure of value. As quality is value, I think the car is high quality.
  • I believe I would get no extra benefit from a Bentley over some other cheaper car, given how I use my car. So I won't pay a high price for a Bentley. This is a measure of value. As quality is value, I think the car is low quality.

And then I reflected:
  • I feel that I can hold these kinds of opposing views at the same time without problem (at least in some cases).
  • I speculate that quality can be a relationship between a person and product-attribute rather than a product.
  • My examples are couched in terms of belief rather than actual knowledge (I have never even sat in a Bentley)
  • ... so to Weinberg I've really I've got a statement about perceived value, if anything, here.
  • Perhaps related, quality assessment of wants (I'd love a Bentley) could be different to needs (I have to have some personal transport).
  • Is there always, ultimately, some overriding single attribute of quality that wins out for any given person, at a given time, and so multiple perceived qualities collapse at the point of use into a single assessment?
  • Or perhaps simultaneity is a false perception here. Maybe I am switching between views — very rapidly — and only hold one at any given time.
  • Another angle: when I consider two contexts of use, or aspects, or applications of a product, could really be considering effectively two different products?

I tried another scenario, which attempts to take belief and perception out of the equation by using a more mundane product that I have personal experience of. Let's say I have bought a new pen and I want to use it for two tasks: taking notes while standing up and taking notes while suspended by my feet.
  • The pen is suitable for the first task and I am very happy with the price I paid. I say this is a good quality pen.
  • The pen is not suitable for the second task. I had to keep inverting it to let ink run back to the nib end, which I am unwilling to do any longer. I say this is a low quality pen.

More thoughts:
  • At the point where I pay for my pen, by Weinberg's model, I make an explicit statement about the quality of the pen for me.
  • Unintuitively, perhaps, if I've never used such a pen before this is based only on my perception of the value the pen will return to me
  • ... so perceived quality can turn into actual quality with no additional evidence to back it up
  • ... and, on engagement with the pen, I might rapidly revise my opinion.
  • Once I've paid for it, I express my view of the quality of the pen by the extent to which I am prepared to sacrifice to use it
  • ... and (if I understand the model) effectively the only time at which I can express this view is at the point of use
  • ... because at other times I merely express a perception of what I would do when I came to use it
  • ... and so can I change my expression of the quality of the pen without using it?
  • For example, can I express an opinion on quality by choosing not to use something?
  • ... but then how to distinguish between something that I happen not to use and something that I actively don't use, and something that I use only occasionally but is perfect for a particular task?

And then I stopped and dumped my notes here, after pondering how much I was prepared to sacrifice to continue this particular line of thought at this time.

With thanks to Jerry for patiently listening to me trying to make some kind of argument along these lines in email, and then patiently declining to agree. And also to Šime for prompting more thoughts when I was going round in circles.

Wednesday, June 28, 2017

Cambridge Lean Coffee

This month's Lean Coffee was hosted by us at Linguamatics. Here's some brief, aggregated comments and questions  on topics covered by the group I was in.

If we don't do testing, what do we replace it with?

  • We move test environment and tooling into Dev.
  • But practically, how do you ensure the customer gets the right thing?
  • Testing vs checking: testers need to exist.
  • Perhaps the tester just becomes an advisor?
  • With more ability to push into production more often and roll back if there's a problem, there can be less testing.
  • Even if testing is done elsewhere (by developers or customers) we still need someone to ask pertinent questions about the product, to evaluate risks.
  • And where is the test manager?
  • The test manager is taking a more strategic view, coaching, keeping people aligned, across products and projects.
  • Testing is being pushed left (into Dev) and pushed right (into production) and up (into the business).
  • Then what would be down?
  • Why do we need test managers? Why not just engineering managers?
  • Managers with relevant technical skills are respected by staff.

Formal test plans. How can they help in coordinating phases or levels of development?

  • A fair analogy to the questioner's context might be coordination between teams building layers of unit tests, integration tests, end-to-end tests.
  • How do you get the right amount of co-ordination between the different phases?
  • How can you compare the coverage in each phase?
  • Can a formal test plan (whatever that is) be a way to begin to share what's done in each phase?
  • ... and make it consistent across releases?
  • Talk to people!
  • Consider checklists over something heavyweight.
  • What problem are you trying to solve here?
  • A perception that there is repeated work in phases, and that this impedes delivery to market.
  • Are you concerned that there might be testing that no-one is doing?
  • Oops.
  • Can all phases work in one environment so that more can be shared?
  • Is there a way to instrument environments to tell what of the product functionality is being exercised in each environment?
  • ... perhaps something like code coverage metrics in software?

What are you reading? Has it helped you? How?

  • Conceptual Blockbusting by James L Adams.
  • ... it identifies ways in which creativity is blocked
  • ... and it suggests techniques for overcoming them
  • ... which I find valuable as I view testing as inherently creative.
  • Podcasts more than reading at the moment
  • ... because I can do them at the same time as something else
  • ... topics include culture, science, testing, mental health, sports
  • ... and I find I can draw parallels to my work.
  • Coaching Agile Teams by Lyssa Adkins
  • ... right now, I'm re-reading the section on conflict resolution
  • ... because it's relevant to my work situation.

Monday, June 12, 2017

A Test Manager?

CEWT #4 was about test management and test managers. One of the things that became apparent during the day was how much of a moveable feast the role associated with this title is. And that reflects my own experience.

A few months ago, when discussing courses for the line managers in the Test team, a trainer outlined what his course would cover and asked whether I'd got any heuristics for management. I gave him these, none of which were included in his synopsis:
  • Clear and present. (Say what you think and why, and what you are committed to; encourage and answer any question; be approachable, available and responsive, or say when you can be)
  • It’s all about MOI. (Motivation: explain why we are doing what we’re doing; Organisation: set things up to facilitate work, opportunities; Innovation: be ready with ideas when they’re needed)
  • Congruency in all decisions. (Consider the other person, the context, yourself)

In advance of CEWT, one of my team asked me what I felt my responsibilities as a Test Manager are. Off the top of my head, I suggested they included following:
  • Provide appropriate testing resource to the business.
  • Assist in the personal development of my staff.
  • Develop relationships in my teams, with my teams, across teams.

At the pub after CEWT last night I was asked what I did as a Test Manager. I replied that it's changed a lot over time, but has encompassed situations where:
  • I was the sole tester. (And also learning how to be a tester.)
  • I was planning and scheduling the testing for a small test team, working on a single product. (And also learning about planning and scheduling for others.)
  • I was planning assignments of testers to projects and teams across products. (And also learning about how to work without knowing so much about some of the work my team are doing.)
  • I was managing larger and larger teams. (And learning how to be a better manager.)
  • I was delegating larger and larger projects to other testers. (And learning how to help others to manage larger projects.)
  • I was keeping track of more and more projects across the company, as we grew. (And learning about finding ways to get the right information at the right costs.)
  • I was delegating line management responsibility to other testers. (And learning about how to help others find and express themselves in line management roles.)

Ask a slightly different question, or a different test manager, or in a different context, or about a different time ...

Get a different answer.

Thursday, June 8, 2017

But is it Automation?

Recently, I needed to quickly explore an aspect of the behaviour of an application that takes a couple of text file inputs and produces standard output.

To get a feel for the task I set up one console with an editor open on two files (1.txt and 2.txt) and another console in which I ran the application this way:
$ more 1.txt; more 2.txt; diff -b 1.txt 2.txt
a b c d e f
a b c
d e f
< a b c d e f
> a b c
> d e f

$ more 1.txt; more 2.txt; diff -b 1.txt 2.txt
a b c d e f
a b c  d e f

$ more 1.txt; more 2.txt; diff -b 1.txt 2.txt
a b c d e f
a b cd e f
< a b c d e f
> a b cd e f
As you can see I have a single command line that dumps both the inputs and the outputs. (And diff was not the actual application I was testing!)

After each run I changed some aspect of the inputs in the first console, pressed up and enter in the second console.

What am I achieving here? I have a simple runner and record of my experiments and an easy visual comparison across the whole set. It's quick to set up and in each iteration I'm in the experiment rather than the infrastructure of the experiment.

I could have, for example, created a ton of files and run them in some kind of scripted harness or laboriously by hand. But I was short of time and I wanted to spend the time I had on exploring - on responding to what I'd observed - and not on managing data or investing in stuff I wasn't sure would be valuable yet.

I still hear and see too much about manual and automated testing for my comfort. Is what I did here manual testing? Is it automation? Could a "manual tester" really not get their head around something like this? Could an "automation tester" really not stoop so low as to use something this unsophisticated?

Bottom line for me: there's a tool that is at my disposal to serve my needs at appropriate cost, with appropriate trade-offs, and in appropriate situations. Why wouldn't I use it?
Syntax highlighting:

Friday, June 2, 2017

Taking Note

In Something of Note, a post about Karo Stoltzenburg and Neil Younger's recent workshop on note-taking, I wrote:
I am especially inspired to see whether I can distil any conventions from my own note-taking ...  I favour plain text for note-taking on the computer and I have established conventions that suit me for that. I wonder are any conventions present in multiple of the approaches that I use?
Since then I've been collecting fieldstones as I observe myself at work, talking to colleagues about how they see my note-taking and how it differs from theirs, and looking for patterns and lack of patterns in that data.


I already knew that I'd been revising and refining how I take notes on the computer for years. Looking back I can see that I first blogged about it in The Power of Fancy Plain Text in 2011 but I'd long since been crafting my conventions and had settled on something close to Mediawiki markup for pretty much everything.  And Mediawiki's format still forms the basis for much of my note-taking, although that's strongly influenced by my work context.

These are my current conventions for typed notes:
  • * bullet lists. Lots of my notes are bullets because (I find) it forces me to get to "the thing"
  • ... as a way to carry on thoughts across bullets while preserving the structure
  • > for my side of a conversation (where that is the context), or commentary (in other contexts)
  • / emphasis
  • " for direct quotes
  • ---- at start line and end line for longer quoted examples, code snippets, command line trace etc
  • ==, ====, ==== etc for section headers
  • +,-,? as variant bullet points for positive, negative, questionable
  • !,? as annotations for important and need-to-ask

These are quick to enter,  being single characters or repeated single characters. They favour readability in the editor over strict adherence to Mediawiki, e.g. I use a slash rather than  repeated single quotes for emphasis because it looks better in email and can be search-replaced easily.

I am less likely to force a particular convention on paper and I realise that I haven't put much time into thinking about the way I want to take notes in that medium. Here's what I've come up with by observation:
  • whole sentences or at least phrases
  • quotation marks around actual quotes
  • questions to me and others annotated with a name
  • starring for emphasis
  • arrows to link thoughts, with writing on the arrows sometimes
  • boxes and circles (for emphasis, but no obvious rhyme or reason to them)
  • structure diagrams; occasional mind map
  • to-do lists - I rarely keep these in files
  • ... and I cross out what I've done
  • ... and I put a big star next to things I crossed out that I didn't mean to

Why don't I care to think so hard about hand-written notes? Good question. I think it's a combination of these factors: I don't need to, I write less on paper these days, the conventions I've evolved intuitively serve me well enough, it is a free-form medium and so inventing on the fly is natural, information lodges on paper for a short time - I'll type up anything I want to keep later.

Similarities and Differences

I want to get something of that natural, intuitive spirit when typing too, although I'm not expecting the same kind of freedom as a pen on paper. What I can aim for is less mediation between my brain and the content I'm creating. To facilitate this I have, for example:
  • practised typing faster and more accurately, and without looking at my fingers
  • learned more keyboard shortcuts, e.g. for navigating between applications, managing tabs within applications, placing the cursor in the URL bar in browsers, and moving around within documents
  • pinned a set of convenient applications to the Windows taskbar in the same order on all of the computers I use regularly
  • set up the Quick Access Toolbar in Office products, and made it the same across all Office products that I use
  • made more use of MRU (most recently used) lists in applications, including increasing their size and pinning files where I can

With these, for example, I can type Windows-7, Alt-5 to open Excel and show a list of recently-used and pinned files. Jerry Weinberg aims to record his fieldstones within five seconds of thinking of them. I don't have such strict goals for myself, but I do want to make entering my data as convenient as possible, and as much like simply picking up a notepad and turning to the page I was last working on as I can.

That's one way I'm trying to bring my hand and typed note-taking closer together in spirit, at least. There are also some content similarities. For instance, I tend to write whole sentences, or at least phrases. Interestingly, I now see that I didn't record that in my list of conventions for typed notes above. Those conventions concentrate solely on syntax and I wonder if that is significant.

I don't recall an experiment where I tried hard not to write in sentences. The closest I can think of is my various attempts to use mind maps, where I find myself frustrated at the lack of verbal resolution that the size of the nodes encourages - single words for the most part. Again, I wonder whether I don't trust myself enough to remember the points that I had in mind from the shorter cues.

In both hand and typed notes, I overload some of the conventions and trust context to distinguish them. For example, on paper I can use stars for emphasis or specifically to note that something needs to be considered undeleted. On screen I'll use ? for questions and also uncertainty. I also find that I rarely start numbered lists because I don't want the overhead of going back and renumbering if I want to insert am item into the list,

Something else that I do in both cases is "layering". In Something of Note I mentioned that I'd shown my notes to another tester and we'd observed that I take what I've written and add "layers" of emphasis, connections, sub-thoughts, and new ideas on top of them. (Usually I'll do this with annotations, or perhaps sidebars linked to content with arrows.)

Similarly, one of my colleagues watched me taking notes on the computer during a phone call and commented on how I will (mostly unconsciously) take down points and then go back and refine or add to them as more information is delivered, or I have commentary on the points I've recorded.

There are some differences between the two modes of note-taking. One thing that I notice immediately is that there is no equivalent to doodling in my computer-based notes where my hand-written notes are covered in doodles. I don't know what to conclude from that.

Also, I will use different textual orientations in my written notes, to squeeze material into spaces which mean it is physically co-located with text that is related to it in some way. I don't have that freedom on screen and so any relationships have to be flagged in other ways, or rely on e.g. dynamically resizing lists to add data - something that's less easy on paper.

Where I am aggregating content into a single file over time - as I do with my notes in 1-1 meetings - I almost always work top-down so that the latest material is at the bottom and I can quickly scroll up to get recent context. (I find this intuitive, but I know others prefer latest material at the top.)

Because I don't aggregate content over time in the same way on paper, I don't have quite the same option. I write all of my notes into the same notebook, regardless of context (though I may start a new page for a new topic) so I don't have lots of places to look for a particular note that I made.

Within a notebook, I can flick back through pages to look for related material. I date-stamp my notebooks with a sticker on the front so that I can in principle go back to earlier books, but I rarely do either over periods anything longer than a handful of days.

One other major difference - a side-effect, but a significant one - is that I can easily search my computer notes.


I found that there are situations where I'll tend to use one or other of the note-taking techniques, given free choice. I prefer hand-written notes for:
  • technical meetings
  • meetings where it's less important that I maintain a record
  • meetings where typing would be intrusive or colleagues have said they find it distracting
  • informal presentations, our Team Eating brown bag lunches, local meetups
  • face-to-face job interviews
  • team meetings
  • to-do lists
  • when I need to make diagrams
  • when I don't have access to my computer

Whereas computer-based notes tend to be used for:
  • 1-1 (whether I'm the manager or the report)
  • writing reports
  • writing testing notes (including during sessions)
  • writing blogs
  • where I'm trying to think through an idea
  • when I want to copy-paste data from elsewhere or use hyperlinks
  • when I want to not have to write up later
  • when I want to be able to continue adding content over an extended period of time 

And there are occasions where I use both in tandem. For example, when engaged in testing I'll often record evidence in screenshots and drop the file location into my notes.

I might sketch a mind map on paper to help me to explore a space, then write it up in an editor because that helps me to explore the nature of the relationships.  This is probably a special case of a more general approach where I'll start on paper and switch to screen when I feel I have enough idea - or sometimes when I don't - because editing is cheaper on the computer. From Tools: Take Your Pick:
Most of my writing starts as plain text. Blog posts usually start in Notepad++ because I like the ease of editing in a real editor, because I save drafts to disk, because I work offline ... When writing in text files I also have heuristics about switching to a richer format. For instance, if I find that I'm using a set of multiply-indented bullets that are essentially representing two-dimensional data it's a sign that the data I am describing is richer than the format I'm using. 
In particular, I will aggressively move to Excel for tabular data. (And I have been refining the way I use Excel for quick one-off projects too; I love tables.)


I am an inveterate note-taker and I think I'll always prefer to record more rather than less. But when it comes to the formatting, I'll always prefer less over more. For me, the form should serve the content and nothing else, and a simpler format is (all other things being equal) a more portable format.

It appears that I'm happy to exploit differences where it serves me well, or doesn't disadvantage me too much - I clearly am not trying to go to only hand-written or only computer-based notes. But I do want to reduce variation where it doesn't have value because it means I can switch contexts without having to switch technique and that means a lower cost of switching, because I might already be switching domain, task, type of reasoning etc. In a similar spirit, I am interested in consolidating content. I want related notes in the same place by default.

But I'm not a slave to my formatting conventions: something recorded somehow now is better than nothing recorded perfectly later. I will tend to do the expedient over the consistent, and then go back and fix it if that's merited. I very deliberately default to sticking to my conventions but notice when I find myself regularly going against them, because that indicates that I probably need to change something.

Right now I am in the process of considering whether to change from ---- at the start and end of blocks to using three dashes and four dashes at start and end respectively. Why? Because sometimes I need to replace the blocks with <pre> and </pre> tags for the wiki. Marking up the start and end with the same syntax doesn't aid me in search-replacing.

When I am trying to introduce some new behaviour, I will force myself to do it. If I fail, I'll go back and redo it to help to build up muscle memory. I think of this as very loosely like a kata. For example, I was slower at typing for a while when I started to type in a more traditional way, but I put up with that cost in the belief that I would ultimately end up in a better place. (And I did.)

I think that my computer note-taking is influencing the way that I write non-note content. To give one illustration: over the years I have evolved my written communications (particularly email) to have a more note-like structure. I am now likely to write multiple one-sentence paragraphs, pared back to the minimum I think is necessary to get across the point or chain of reasoning that I want to deliver.

Likewise, I try to write more, shorter paragraphs in my blog, because research I've read, and my own experience, is that this is a more consumable format on screen.  (After seeing how much content I'd aggregated for this blog post, I considered splitting it too.)

I use text files as repositories of related information, but I also sometimes have a level of organisation above the file I'm working in. I'm recruiting as I write this. If, after I review a CV, I want to talk to the candidate, I start a text file in the folder I'm maintaining for this round of recruitment. My notes on the CV go there, as do questions I'll ask when we speak. On the phone I'll type directly into the file, recording their answers, my thoughts on their answers, new questions I want to ask and so on. At the end of the interview, I'll briefly review and note down my conclusions in the file too.

The same technique applies to my team. I have weekly 1-1 with my team and an annual review cycle. I make a folder per person, inside that a folder per cycle and, inside that I have a text file, called Notes.txt. In 1-1 I will enter notes while we talk. Outside of 1-1 I'll drop thoughts, questions, suggestions and so on into the file in preparation for our next meeting. Over time, this becomes an historical record too, so I can provide longitudinal context to discussions.

This stuff works for me - or at least, is working for me right now better than anything else I've tried recently and given the kinds of assessments I've made of it - but none of it is set in stone. My overarching goal is to be efficient and effective and I'm always interested in other people's conventions in case I can learn something that helps me to improve my own.

Sunday, May 28, 2017

Best Practise

I've said many times on here that writing for me is a kind of internal dialogue: compose a position, propose it in writing, challenge it in thought, and repeat.

I get enormous value from this approach, and have done for a long time. But in two discussions last week (Lean Coffee and a testing workshop with one of the other teams at Linguamatics) I found additional nuances that I hadn't considered previously.

First: in some sense, the approach I take is like pairing with myself. Externalising my ideas sets up, for me, the opportunity to take an alternative perspective that doesn't exist to the same extent when I'm only working in my head. It's often about the way I'm thinking as much as the content of my thoughts, and I speculate that this is a good grounding for being criticised by others when we're working together.

Second: writing and re-reading makes my position clear to me, and forces me to work out a way in which I can put it across. Since I started blogging there are numerous times in discussions that I've realised I am paraphrasing from something I've written. In the past I've tended to be a bit embarrassed by that but now I can see that, in fact, it's largely because I spent the time working it out before that I have it available to me now.

These are both things that are useful to me and that I want to get more benefit from. And, while I might agree that outside of a specific context there are no best practices, I also know that if I want to get those outcomes from my writing, I'd best practise.

Wednesday, May 24, 2017

Cambridge Lean Coffee

This month's Lean Coffee was hosted by Redgate. Here's some brief, aggregated comments and questions  on topics covered by the group I was in.

What benefit would pair testing give me?

  • I want to get my team away from scripted test cases and I think that pairing could help.
  • What do testers get out of it? How does it improve the product?
  • It encourages a different approach.
  • It lets your mind run free.
  • It can bring your team closer together.
  • It can increase the skills across the test group.
  • It can spread knowledge between teams.
  • You could use the cases as jumping-off points.
  • I am currently pairing with a senior tester on two approaches at the same time: functional and performance.
  • For pairing to work well, you need to know each other, to have a relationship.
  • There are different pairing approaches.
  • How long should you pair for?
  • We turned three hour solo sessions into 40 minute pair sessions.
  • You can learn a lot, e.g. new perspectives, short-cuts, tips.
  • Why not pair with developers?

Do you have a default first test? What it is? Why?

  • Ask what's in the build, ask what the expectation is.
  • A meta test: check that what you have in front of you is the right thing to test.
  • It changes over time; often you might be biased by recent bugs, events, reading etc to do a particular thing.
  • Make a mind map.
  • A meta test: inspect the context; what does it make sense to do here?
  • A pathetic test: just explore the software without challenging it. Allow it to demonstrate itself to you.
  • Check that the problem that is fixed in this build can be reproduced in an earlier build.

How do you tell your testing story to your team?

  • Is it a report, at the whiteboard, slides, a diagram, ...?
  • Great to hear it called a story, many people talk about a report, an output etc.
  • Some people just want a yes or no; a ship or not.
  • I like the RST approach to the content: what you did, what you found, the values and risks.
  • Start writing your story early; it helps to keep you on track and review what you've done
  • Writing is like pairing with yourself!
  • In TDD, the tests are the story.

One thing that would turn you off a job advert? One thing that would make you interested?

  • Off: a list of skills (I prefer a story around the role).
  • Off: needing a degree.
  • Interested: the impression that there's challenge in the role and unknowns in the tasks.
  • The advert is never like the job!
  • Interested: describes what you would be working on.
  • Off: "you will help guarantee quality".
  • Interested: learning opportunities.
  • Interested: that it's just outside of my comfort zone.