A Beginner’s Guide to Tracking (Longitudinal) Research

As always, I am presenting from within Display, as it's designed for interactive presentations. but everything I show, can be done in Q as well. So, if you are a Q user and there are bits you want me to show you in Q, just tell me and I will do it at the end.

Tracking research...

Tracking research involves asking the same set of questions at different points in time. Typically, to a different group of people at each time period.

The idea is to track what changes in he market.

In some markets continuous tackers conduct inerviews every day, week, or month. However, dip sticks, either regular or irregular, are the norm in many markets.

Challenges

Tracking research involves solving challenges that we can put into three buckets:

• Data management
• Analysis
• Reporting

Data management

Tracking studies have four big data management challenges.

The first is changes in the questionnaire.

1. New questions
2. New brands
3. New advertisements

A second is clerical errors. Working on tracking studies can be boring. Doing the same thing again and again. Mistakes happen.

The third problem is how you categorize the text data. Or, to use the market research jargon, how you code it.. No two people read text the same way, so if you change who does the text categorization, you can end up with patterns in the data over time that just relate to who performed the cateorization.

And, the fourth big issue is that all the above increase the time and costs of reporting.

So, how do we solve these problems.
It's easier said but done, but the biggest secret is to fight tooth and nail to avoid making unnecessary changes to a tracking questionnaire.

The second point is to work even harder than usual to get a good data file format.

Good file formats likes SPSS and Triple S files contain lots of metadata, such as question wordings, and the order in which options were presented.

If doing a tracking study, changes to the questionnaire are easy to detect and address, if you have these better file formats.

By contrast, when you are using CSV and Excel files, you can make changes to how you collect the data, but have no way of finding these in the raw data, which leads to mistakes.

I'll talk about the next few points. But, a quick notice that more advanced topics, shown in blue, will be covered in the next webinar

Use cumulative ...

OK, let's look at the wrong way to set up your tracker.

You first interview everybody in the first wave.

Then, you copy the project in your data collection software and send the new version off to a second group of people for your second wave.

If you do that, you end up with two data files.

This has two problems.

First, you will need to merge them.

That's usually pretty easy o do.

In Displayr, for example:
In Display: Combine > Data Sets > Merge by Case

But, the bigger issue will be that when you merge them, you will inevitably lose information about how the questionnaire has changed over time.

The smart way forward is to only have the one data collection project that contains all the respondents. So, when you do analysis, you just export all the data in a single file. This is known as a cumulative data file.

Automate as much as you can

I will go into a lot more detail on automation in the advanced webinar. However, in this webinar I will just focus on the ideal end goal.

The end goal of your automation work is that your software, rather than you, do all the automation. This means fewer errors, faster turnaround, and lower costs.

Let's look at an example.

Here's a dashboard. It's for the data we've just been looking at. It's showing us the results for July 2015 to September 2017.

Burger consumption in the past month is at 83%, which rose 4% from the previous month.

Now, I'm going to replace my data file, with a new cumultative file, that contains all my old data and one more quarter.

Update data file

It' will take a moment It's got a lot of thinking to do.

Now, remember, our score was 83%, and this has dropped 78%. For

Everything else has automatically updated. The raw data, the underlying tables, calculations, and all visualizations.

That’s where you can get to with automation. But it’s a lot of work. We will return to it in the advanced webinar

Analysis

When we are doing analysis, our key issues are:

• Finding meaningful changes.
• Avoiding reporting noise as meaningful
• Not confusing trend with seasonality

There are three sets of tools for solving these problems

• Automated stat testing
•  Weighting
• And, using our brains to attempt to work out causality.

Let's dig into theses.

Stat testing over time

We have a set of key questions. For example, here's some data on aided awareness.

We want to look at it over time.

If we have a dip stick, the data may be categorical, and we create a standard crosstab.

The goal of stat testing is to try and help us work out which results aren't just noise.

If you are a true beginner, you might look at this and think. Ah, Arnold's declined in Wave 5 from Wave 4.

But, that's not right.

By default, stat testing is not set up for trackers. Instead, by default, stat tests compare each period with all the other periods, rather than just the one before.

So, how do we rectify this?

One approach is to just delete other columns.

Delete Wave 1 through 3

Now, we can see that the Arnold's difference is no statistically significant. Rather, Bread Basket and Nuovo burger changed.

Or, we can use column comparisons.

Properties Significance > Column comparisons.

So, the previous wave, wave 4, is in column d. So, we look in the Wave 5 column for ay Ds, and, just as we saw before, we can see it only is different for Bread Basket and Nuovo burger.

If we have continuous tracking or regular time periods, we will want to do our analysis with a variable that contains the actual date of the interview.

In Displayr and Q, this will mean we have the data set up as a date variable, which will mean we will see a calendar icon like this.

Displayr's automatically grouped the time into monthly buckets. I've got too much data to see.

So, to simplify the data we will aggregate it. This is good practice. If you have too much data, simplify it.

Let's change that to Quarterly.

Interview date > Date/Time
Aggregation: Quarterly

When we are looking at tracking data, our main focus is always on the very last period, which is Jul-Sep 17.

We're still showing the column comparisons. With this many date periods they'r a pain, and we're better off showing exception tests.

We need to do two things:

First, we turn on the exception tests

Significance > Arrows and Font Colors

As before, these are comparing all the time periods. Note that we are seeing Burger Shack is up. But, this is telling is that Jul-Sep 17 is higher than the rest of the data, rather than just comparing to he previous period. So, it's the same beginner's mistake from before, and we need to tell the software to compare to the previous period.

Significance > Advanced > Compare to

And look, we have a completely different conclusions. The only significant difference is now Pret'a'pane.

Often we want to compare data to where things were at 12 months ago. Here's a couple of reasons for this

Sometimes, it's jus an interesting statistic to look at.

However, in some markets there are strong seasonal effects, and certain brands are stronger in, say, summer. In such markets, we're more interested in comparing to a year ago.

Looking at Burger Chef in the first row, since July September 16, i's gone up from 74% to 71%. Is this statistically significant?

Stat testing: comparing to 12 months ago

But what if we want to compare to a year ago?

I'm going to do this by creating a new variable

+ > Insert Read-Made New Variable(s) > Contrast Periods

Let's drag it across and look at it.

It's showing the last 2 months of the data

So, i'll tell it to focus on quarters

Date aggregation: Quarter

It's now showing the same result we had before for Pret-a-pane. But, we can now control where we compare against.

Versus: Q3 2016

OK, so we can see that quite a few brands are up compared to a year ago.

With such data, it's often nice to show a difference.

In Displayr, we can do this with a Rule.

Rule > Tabe Computations > Difference Between Pair of Columns
Column 1: Q3 2017
Column 2: Q3 2016

And, commonly people may just want to look at the difference.

In this example, we are getting quite different conclusions when comparing to 12 months ago than when just comparing to the previous time period.

So, we need to understand the trend. We will look at the stat testing side of this in the advanced webinar.

But, the key thing is to create a good way of showing all the data, which we will come to shortly.

Weighting is almost always appropriate in tracking studies

In all studies, we get over- and under-representation of different sub-groups. For example, in a tracker, we may have too many men one month, and two few the next.

Weighting can correct for this.

With a tracker, we can expect that every wave we will get such random movement.

Therefore, we pretty much always want to weight a tracker.

Here's some data over time.

Lucky's Pizza shows no significant differences.

What happens when we weight it?

Now we are seeing a difference. This is interesting. The random noise had hidden a pattern, and it's only revealed when we apply the weight.

So, remember, if you can, you should weight your trackers.

We will talk about some more advanced aspects of weighting in our next webinar on tracking.

And, check out our webinar on weighting for more about weighting in general.

Analysis+

Automated stat testing and weighting both help us to sort out the true differences from the random noise.

These should be augmented with attempts to understand causality. That is, trying to figure out what has caused a movement. If you can find other data that that corroborates the differences you see in the tracker, that's a good sign. Examples of other data to look at are:

Marketing activities, such as changes in ad campaigns and distribution.

• Sales data
• Other surveys
• Google rends.

And, the last thing to do is to decompose changes over time in terms of whether they are driven by changes within sub-groups or changes in sub-group size. There's a nice framework and process for this, which will be the centrepiece of our more advanced webinar on tracking.

Reporting

When we report our tracking, our first key challenge is making it easy for our audience to spot meaningful changes in the data .

Our second challenge is to stop them from seeing patterns in noise.

And, the biggest challenge is stopping them from being bored!

Two great tools for doing this are small multiples and smoothing. We will dig into these shortly.

Another great tool is to have rotational sections of your studies and reporting. E.g., one month maybe you focus on brand health. A second month ad effectveness, and so on.

One particular variant of rotating deep dives is to identify laws of the market, which we will focus on in the advanced webinar.

Spaghetti

OK, so every now and then I see a visualization that's just so horrible I had to take a screen shot.

To prevent embarrassment from all concerned, I've hidden the brand names. It's not a good chart.

In interactive..

This is another line chart showing tracking data. Because it's inteactive it's a bit better.

But, to find a pattern we need to work really hard. Tell me, for example, what's the story with The Crown? What pattern can you see?

Yep, nothing really.

The awesomely named mathematician, Lofti Zadah, had a saying: When the only tool you have is a hammer, everything begins to look like a nail.

And, in the world of PowerPoint and Excel, the line charts like these are the only hammer for time series data.

Fortunately, there's a much better solution, which is small multiples. But, you can't do them in Excel.

"Small multiples"

But, we're not in Microsoft anymore toto, so let's do something better.

Now, have a look. What's the story with the Crown? We can now make out lots of really clear patterns in the data.

Brand awareness spaghetti

Here's our earlier brand awareness data. It's not as much as a mess as the earlier line chart.

Brand awareness small multiples

But the small multiple are still so much better!

Smoothers

Small multiples are the number 1 data visualization tool that you need to know about for tracking.

The number 2 is to use smoothers.

Here's some data on customer churn. It moves around a lot.

We could aggregate to show it quarterly. Or, we could put a trendline on it.

Object inspector > Chart > TREND LINES > Line of best fit : Line

Clearly a straight line tells us something, but it's not doing a great job.

Many make researchers use rolling averages, also known as moving averages to show the underlying trend.

Object inspector > Chart > TREND LINES > Line of best fit : Moving Average

For example, this one is a 6 month moving average.

It's called a smoother, because the resulting trend line is a lot smoother than the original data. The idea is that the smooth data will show make it easier to see the trend.

Unfortunately, this rolling or moving average isn't doing a great job. Notice hat:

We've got no line for the first five periods.

And, notice also that the line doesn't do a great job at showing trends. For example, in the raw data our two big peaks are here and hear

But, the peaks occur in the moving average well after this. What's going on?

Well, this ones a bit sad. Moving averages are one of those things that get taught in undergraduate stats as an introductory topic, to help people understand. But, they're not meant to be used!!!

Much, much better techniques have been invented, such as

• LOESS
• Object inspector > Chart > TREND LINES > Line of best fit : LOESS
• And Friedman's super smoother
• Object inspector > Chart > TREND LINES > Line of best fit : LOESS
• Notice how this does a much better job.

Which of these more advanced techniques should you use? Whichever one does a better job at describing the data.

How do they work?

Well, they are better, but they are really complex. So, I'm not gonna even try.

And, what's wrong with the rolling average?

Let's do a worked example.

Why rolling/moving averages are always misleading

We're going to do some very simple math. Do try and follow along, as if you are using rolling averages, I what I will show you is really interesting and surprising.

Let's look at a 6 month rolling average

We calculate the average for October 2019 as the average of the six numbers shown here.

As you hopefully remember, this is the formula for the average.

+

And, the average for November is calculated this way.

++

So, the October number is 6, and November is 6.4. That suggest that once we smooth the data, churn's gone up by 0.4

+++

OK, Now a bit of simple maths.

6.4 - 6

Is the same this line. That is, I've just substituted

that needs to be used. The number 2 is modern smoothers.

Here' I've got some data on customer churn.

Lets plot this data as columns

Insert > Visualization > Column Chart

It jumps around a lot. 99 out of 100 market researchers then go and apply a moving average, also known as a rolling average.

Chart > Trend > Moving average

Displayr's defaulting to the 3 month moving average. This is wiggling around too much.

Let's try 6 months

But, look, the moving average actually does a bad job. Look in June 2019. A huge spike. The moving average shows a small blip up.

According moving average, the data peaked in November 2019. But, look at the actual data. The high points are 2 and 5 months earlier.

What's going on?

The sad truth is that traditional moving averages are just dumb. Statisticians have known for more than 100 years that they are dumb. This is why whenever you see government data or anything done by a good statistician, they don't use moving averages.

There are lots of alternatives.

Three of the popular ones are in Displayr and Q.

Line of best fit: LOESS

One of them is called Locally Weighted Exponential scatterplot smoothing, or loess for short.

It's doing a much better job. It works out that the June 2019 peak was an aberration. And, it picks that since September things have been getting worse.

Another one, with a cooler name, is Friedman's super smoother

Line of best fit: Friedman's super smoother

Or, you can have a cubic spline.

Which is best? Just like with choosing wether to do a 3 or 6 month moving average, it comes down to the story you want to tell.

These techniques are always better than moving averages. Always is a strong word. And I mean it. If your goal is accurately summarize the data, these techniques are provably always better than a moving average.

Nothing comes for free however.

The math is hard. You need to point, click and trust. You are unlikely to ever understand the math. And, unlike a moving average, when you get new data, the historical line will update, so they're not great for setting performance target for bonuses.