LOD equivalent of LOOKUP

The LOOKUP table calculation in Tableau is really handy when you want to show or use a value from a previous row in the view. For example if you are showing sales per month and need to use the sales figure from the previous month to calculate month-on-month growth.

Sales and sales previous month


In the example above our calculation for “Sales last period” is:

LOOKUP(SUM([Sales],-1))

LOOKUP is great when the level of detail you want to look back through is in the view (e.g. order month is in the view if you want to lookup sales for the previous order month). But things get tricky when that isn’t the case. And people often hit constraints when they want to use the result of a table calculation like LOOKUP in another calculation / aggregation. It’s not unusual at this point for people to ask if they can use a level of detail / LOD calculation for looking up the value from a previous month, year, or category. And unfortunately the answer is almost always “you can’t use a LOD for that”, but can you? Spoiler alert: yes!

The trouble with LODs

LOD calculations allow you to break out of the level of detail that your calculation is working on. So if your calculation is working at the month level you can break out to the quarter level, to ask questions like how does this month compare to the quarter. In the example below we could use a LOD to calculate how much the sales for 2022-06 differ from the average for the quarter that that month is in;  breaking out to a higher level grouping.

Sales by month and quarter highlighting a month and quarter


But if you’re after previous month there is no single dimension or higher level grouping that will tell you that; each month has a different previous month (unlike the case above where three months share the same common quarter). The previous month for 2022-06 is 2022-05, but 2022-06 is itself the previous month of 2022-07. Any one month is in two groups when it comes to determining previous months.

Possibly as you read that last sentence an idea will be forming! Let’s pursue that idea that there are two groups in play…

A LOD for LOOKUP(…,-1)

If we put each month into two groups which are offset from each other by a month then we can calculate two previous values, one of which will be right for odd numbered months and the other for even numbered months in a sequence of months. Let’s take a look at this in action:

LOD version of LOOKUP, example with monthly sales


Firstly we need an continguous sequence number for each row (in our case each month). With months that is easy as we can use a DATEDIFF to calculate the number of months since the first month in the data set.

_i = 

DATEDIFF(
 'month',
 {MIN(DATETRUNC('month',[Order Date]))},
 DATETRUNC('month',[Order Date])
 )

Using that sequence number we can create the two offset groupings using integer division.

_iGroupA =

DIV([_i]-1,2)

_iGroupB = 

DIV([_i],2)

This gives us a grouping for each combination of two months:

Each grouping of two months


February will be in Group B when we want it’s previous month, and in Group A when it is the previous month.

As we now have a higher level grouping that we can target in a LOD, we create two LOD calculations to get the other value in the group…

_iPrevA =

SUM({FIXED [_iGroupA]: SUM([Sales])})
- 
SUM([Sales])

_iPrevB = 

SUM({FIXED [_iGroupB]: SUM([Sales])})
- 
SUM([Sales])

These calculations get the sum of sales for each grouping of two months, and then subtract the sales for the month we’re in. Hence we end up with the sales for the other month in the grouping. Now we just need to work out which of these LODs to use for each month. To do that we switch between them…

iSwitch =

([_i]-1) % 2

The modulus operator here gives us the remainder of dividing by 2 and hence a number that flicks back and forth between 0 and 1. So…

Sales last month (LOD) =

IF MIN([_iSwitch])=1
THEN [_iPrevA]
ELSE [_iPrevB]
END

You can check out the workbook for this on my Tableau Public profile: LOD equivalent of LOOKUP.

I’ve used this approach for a number of gnarly Tableau Forum questions and I think I’ve seen similar crop up once or twice before but it’s not a particularly well known approach. Often people will fall back on solutions like self joins – and to be fair shaping your data to avoid having to jump through hoops like the above is not a bad idea. But if you do need it, I hope that you find it useful!

What and Why skills vs How skills

John Cutler asked a great question on Twitter; how do we describe less visible skills like qualitative research in comparison to technical skills like software development?

Initially I was intrigued by the parallels with translation vs interpretation in linguistics. I can see similarities between a software developer translating requirements into code. And a design researcher interpreting customer interviews to help produce the right software requirements.

I also liked a response by Tiffany Chang suggesting that one skill is more concrete and the other more abstract. That resonated with human centered design approaches for me. And the idea of not jumping straight from problem to solution:

HCD abstract concrete understand create quadrants



(ref: https://ssir.org/articles/entry/human_centered_systems_minded_design)

So in some ways perhaps we are really talking about the difference between skills to understand “what” (we need to do), and “why” (we should do it), versus skills in “how” we do it? And unfortunately perhaps it is natural for many people to see and appreciate “how” skills above “what” and “why” skills. When you’re cold, fire lighting skills are more immediately appreciable than skills to uncover that we need to move the tribe to a different valley.

Depth vs breadth

In a recent sprint review we were asked how our findings, which were based on a relatively small number of customer conversations, could be meaningful. Were they statistically significant?

I’d got used to our stakeholders being familiar with the background to qualitative research and how we don’t try to quantify it as such. And that the selection approach / recruitment matrix mean that we can have confidence in the insights. However staff had come and gone and so it was a good reminder to address the common concern that a survey would have been better and more statistically significant.

To address the concern we touched on the draw backs of surveys in terms of being sufficiently well designed, depth of understanding and subjectivity. Example – rate our service from 1-10, with a follow up question of why did you pick that rating? If a customer answers “efficient service” what does efficient really mean? Obviously quick and easy right? Well maybe not. I’ve had cases in more in depth conversations where I asked  “Can you tell me a bit more about what efficient meant to you?” and the answer was not what most of us would expect! A survey can you give you breadth but not necessarily depth of understanding. And it’s unlikely to tell you why efficiency is important to customer X.

We also talked about some of the science behind qualitative research: getting your recruitment matrix right, and the concept of saturation and diminishing returns after speaking to a relatively small number of customers. We felt we were close to that point of diminishing returns. And interestingly we’d already identified and highlighted the key drawback of our selection approach in this situation, which was that we were gaining understanding of customer experience across a reasonably narrow time frame – very much “point in time” insights.

Still it is hard for some to trust the insights from what seems like a small number of conversations. My final persuasion is to ask stakeholders to think about some data we have – e.g. the number of times a customer has logged into a help portal. It’s tempting to say that those logging in most are the ones that need the most help and others are doing great without the need for help. But are they really? Maybe they’re struggling on, tapping a colleague on the shoulder and asking for their help instead. The data (in this case) cannot tell you that. A conversation can.

Hone your skills with Makeover Monday

I don’t usually get to attend Tableau User Groups. We don’t (yet?) have one down in the depths of New Zealand’s south island, and it’s a long drive to the nearest one in Christchurch. But with New Zealand and much of the world in some form of lock down, Tableau has encouraged and supported virtual user group meetings. So I was excited to dial into this weeks virtual New Zealand Tableau User Group meeting jointly arranged by Alex, Thabata, Jeff and Paul from the Auckland, Christchurch and Wellington groups. The icing on the cake was being invited to speak about my experience with Makeover Monday!

The topic of my 30 minute slot was Hone your skills with Makeover Monday.  For those after the tips and checklist I mentioned, please read on. Or you can watch the recording of the whole session on YouTube, including the other great speakers:

  • How to do Tableau in lock-down! – Alex Waleczek
  • Hiring: Score yourself a unicorn – Sarah Burnett
  • Set It Up: When to use Set vs Parameter Actions – Heidi Kalbe

Here is the summary of the 13 tips and the example checklist I covered in my presentation.

Tips

  1. First up, my last tip: please, please, please don’t be discouraged from participating by some of the brilliant submissions you’ll see from others. Everyone has to start somewhere. And there’s nothing wrong with a quick and simple makeover. Often a simple bar chart is just what the data and story deserves.
  2. Do read the article, it’s tempting to save time by not reading it. But there is often useful context and background that you can dig into. Spending some time understanding context will usually pay you back as an analyst
  3. Remember the purpose of makeover Monday. Take the time to ask yourself what works well with the original chart and what could be improved. Doing so will help you focus on what you want to make over. Sometimes the improvements might only need to be minor – e.g. better use of colour. Sometime you may be doing a completely different chart.  Invest the time that suits you
  4. Do dig into any nuances in the data. E.g. does the data start and end partway through a year, which would impact seasonal comparisons? Or are there fields that need to be transformed or pivoted to make analysis easier? Have you understood an unusual outlier, or some peaks and troughs?
  5. Once into visual exploration I like to build up working sheets as I explore various angles. That way I can come back to points I want to focus on and refine later. As I refine these, ideas for a dashboard and sequencing start to emerge. My best advice here is to watch one of Andy Kriebel’s live Makeover Monday’s. You’ll get to see where he spends his time, how he goes about exploring data and creating a better data viz.
  6. If you plan to blog about each makeover you’ll find that that takes some time. It can help to keep notes as you review the original chart and explore the data if you plan to blog. That way you can structure them into a blog post at the end
  7. Have a checklist to go through before publishing. Some people keep a written checklist to remind themselves of key things, Otherwise it’s easy to forget about tool tips or spell checking! I’ve included an example checklist below.
  8. Do share your work – take the plunge! It’s a good way to engage and get feedback, which is a crucial part of improving your data visualisation and story telling skills.
  9. Try not get too disheartened if you get no feedback on Twitter, or even unexpected feedback. It’s very difficult to deliver feedback in a way that suits everyone on Twitter.
  10. If you want feedback register for the weekly viz review webinar. Remember to to only use the #MMVizReview hashtag if you will register for and attend the webinar otherwise it makes it harder for the organisers to prepare.
  11. Do work through and incorporate any feedback that you’re given in the viz review webinar. It helps you to reinforce the learnings, and it shows Eva and Charlie that their input is worthwhile!
  12. Don’t be discouraged if you don’t get selected in the weekly favourites. You’re one of a thousand people participating (as of May 2020) and Charlie and Eva can’t practicably see and recall every Makeover Monday tweet! Remember why you decided to take part and ensure that you’re getting what you want out of it – e.g. after two months look back at how much you’ve improved.
  13. If you benefit try to pay it forward in the future. I’ll leave that up to you, but it could be helping new Tableau users on the Tableau Forums (hint: many people find that trying to pass on their knowledge is a wonderful way to gain deeper knowledge themselves), or it could be getting involved in your local user group. Maybe you’ll take the time to encourage new Makeover Monday participants in your area!

Checklist

Here is the example checklist I provided – over time you’ll find the things that help you to check that you’ve got a great makeover before you submit:

  • Right chart type
  • Improved what you set out to improve
  • Remember your audience (e.g. mobile)
  • Clear title & annotations
  • If your title is a question is it answered?
  • Consistent fonts, tool tips, etc.
  • Consistent use of colour (helps the story)
  • Simple is good (remove till you can’t)
  • Spell check and read back through it
  • Source and image credits

Those tips are things that have worked for, or stuck out to me. You can find much more information on the Makeover Monday website, including how to buy the book which covers a whole heap of data visualisation advice.

Finally as I said in the presentation, I look forward to seeing more NZ user group members in the #MakeoverMonday feed soon. Please do reach out if you need some encouragement!

PREVIOUS_VALUE in Tableau

Late last year I started to actively help out on the Tableau Forums. What a great decision! I’d forgotten how much fun it could be to (1) pick up a discrete challenge; (2) help others out; and (3) learn so much more in the process.

One of the questions I recently chipped in on was about circular references in a sequential calculation. The background to the question is really interesting and I ended up spending a few hours digging into epidemiological models, but that’s a different story! Whilst trying to help I took a fresh look at the PREVIOUS_VALUE function in Tableau. I have to admit, prior to this I had thought that PREVIOUS_VALUE(x) was just the same as LOOKUP(x,-1) … turns out that isn’t the case!

LOOKUP(x,-1) will return the previous value of the expression, x, in the “window” of results being presented in the view. So if you’re presenting a monthly sum of [order value], LOOKUP(SUM([order value]),-1) can give you the sum of order value from the prior month*. Handy eh!

PREVIOUS_VALUE(x) is actually saying get the result of this entire calculation from the previous result in the window. And whilst LOOKUP returns NULL on the first result, PREVIOUS_VALUE will return the passed in expression, x, as the first result.

Your initial thoughts might be that these  aren’t much different, but the real power is when you want to do more with your calculation. Let’s say you have:

yA = 2 * LOOKUP(x,-1)

yB = 2 * PREVIOUS_VALUE(x)

In the case of the LOOKUP above you’re always just looking up the last value of x and then multiplying by 2. With PREVIOUS_VALUE you’re starting with x and multiplying by 2, but then each subsequent call is actually looking up the last value of the overall expression – i.e including the  previous multiplication by 2 – then multiplying by 2 again.

So the first yB = 2 * x.

The second yB = 2 * (2 * x) … and so on.

That’s pretty powerful!

The Tableau expression help gives a couple of great examples. The first is to calculate a running sum, and of course you could use RUNNING_SUM for that instead. The second example is pretty cool though – a running product!

PREVIOUS_VALUE_help


I built this viz on Tableau Public to help illustrate the difference in the example I used above. Feel free to download it and have a play.

PREVIOUS_VALUE


Edit:

During subsequent discussions on Twitter Jim Dehner (Tableau Ambassador and top forum contributor) pointed out some additional differences with PREVIOUS_VALUE. I’m yet to dig into these but leave them here in case they help you … “previous value will return the single previous value and can not be chained or used in certain table calculations like running sums -  Lookup is a table calculation that moves from the current “cell” location up, down, left or right by any increment – it can be nested”. I also need to credit Jim for reminding me about the handy example in the Tableau calculation wizard / function help that I included above.

Footnotes:

* with table calculations you get to define how they traverse the window. If you’re displaying a table of months, where each month is a row in your view (i.e. month is on the rows shelf), and you want to LOOKUP the value from the prior month, then you could compute the lookup using “table down”. That way the window progresses down the list of months, and you’re looking up the value from the row above. Table across is often used when you’re calculating a cumulative total on a line graph with time on the horizontal axis (columns). You can define more complex ways to compute the order of the window. And you can specify when to restart the calculation, e.g. to get a separate monthly running total per year.

Makeover Monday, 2019 #26

An interesting and deceptively simple data set on alcohol consumption by country for 2019 week 26.

I like the simplicity of the table of data and the factors affecting the top 25 that are discussed in the article. The chart itself would be better as bars not columns in my opinion, allowing the country names to be laid out for easier reading. As Eva noted in her submission showing liters of pure alcohol consumed per capita per year isn’t that easy to relate to. Digging into the definitions for standard drinks / units I was surprised to find that there is quite a range, and that some countries still don’t define a standard drink. I decided to focus on that aspect for my makeover.

Interactive version on Tableau Public: here.

DASH

Makeover Monday, 2019 #3

Andy Kriebel selected a data set about US workers paid at/below the minimum wage for those choosing to participate in week 3, 2019.

The original viz highlights some of the regional differences for 2017 by showing the data geographically. I like that I can see regional differences, but I found myself wanting to see the trend over time (as it’s available in the data set) to see if the geographical trends are part of an ongoing story.

So for my makeover I’ve kept things pretty simple and separated the different regions and sub-regions. Adding the overall line for the US and differentiating values above / below this in different colours helps to tell the story. A state highlighter allows users to focus in on one state if they want to – this is quick built in functionality for Tableau (right click a dimension and set as highlighter). I spent a lot of time in the depths of SQL Server geography queries for last week’s makeover, so it was refreshing to step back to simple built in Tableau functionality for week 3!

Interactive viz: here on Tableau Public.

Static image:

US-MIN-WAGE

 

Makeover Monday, 2019 #1

Makeover Monday 2019 week 1 looks at NHL attendances since the 2000-01 season.

A couple of things emerge from an exploration of the data set provided: firstly there are seasons where labour disputes, or lockouts, dramatically affect attendances. Secondly some teams have different stories to the general trend. I spent most of my time exploring and presenting the lockout story, but added a team selector to allow users to explore average game attendance by team.

Interactive version on Tableau Public is available here.

DASH

Leading with questions

I was preparing for our company celebration of CX Day 2018  on Tuesday and was reminded of this great interview with Warren Berger on the IDEOU site. The interview drills into the power of questions, and how the right question can lead to a breakthrough and real innovation. The bit that sticks out for me is the question that led to the Polaroid instant camera:

One of my favourite questions is the question that led to the Polaroid Instant Camera back in the 1940s. The four-year-old daughter of the founder of Polaroid asked: Why do we have to wait for the picture? One of the reasons why four-year-olds are good at asking questions is because they don’t have a lot of assumptions and they look at things with a beginner’s mind … Many things begin with a question. It’s this catalytic force. When you arrive at an interesting question and take ownership of that question, it can lead you to innovation.

Brilliant!

My take away: when you’re leading a group of creative people, you don’t need all of the answers; you don’t need your assumptions or your preconceived options for a solution; often you just need a really great question!

Copy and paste text boxes in Tableau

Christina Gorga recently commented on Twitter that she would love the ability to copy or duplicate text boxes on Tableau dashboards.

The tweet attracted favourable attention, with 44 likes. One reason the feature is seen as useful is that it could reduce the time taken to copy formatting throughout a dashboard; styling like fonts, sizes, colours, borders. How much of a pain is it to reapply these to multiple text boxes?

The good news is that there is a Tableau feature request (idea) to copy and paste objects in a dashboard, and we can vote for that to try to get it onto the product roadmap! Like any product development team I’m sure Tableau have to prioritise their investment, and up voting ideas gives them an idea of what to focus on.

In the meantime, if you’re willing to take some risks in a non-critical Tableau dashboard, there is already a way to copy and paste text boxes. I’ve seen this idea mentioned on the forums by Andy Cotgreave, and he quite rightly points out that it is likely to be unsupported. So if you’re going to try it, then take a back up of your workbook first. I’ll tell you more about why this is important towards the end of this post! For now trust me and take a backup.

Right let’s work through the how to guide. Please do excuse the awful dashboard design; it’s purely to illustrate the approach.

How to copy and past text boxes in Tableau

Note that this approach is for floating layouts!

Step 1 is to setup an outline dashboard and add your template text box – this is the text box format you want to copy throughout. In this example it’s the text in the top left and I’d like to duplicate the style in the bottom left and bottom right.

copy-paste-01



Step 2 is to close your workbook and open up the .twb in a text editor (like notepad) instead of Tableau Desktop. The file is mainly metadata about how to transform and display the actual data in your data sources, and is encoded in XML. XML is generally human readable and, importantly to us in this case, human editable. Once open in a text editor find the section that starts “dashboards”. Within this section you should find a section for your particular “dashboard”. I’ve highlighted the section for my dashboard below:

copy-paste-02



I’ve also added some annotations to draw your attention to two parts. A is a zone of type=”text”. As you can see part way down and to the right, it includes the text that I placed at the top of my dashboard. You can see other layout and formatting elements and attributes in this overall section like x and y coordinates, height and width and some styling. One aspect not included here is text alignment. You can see that in the section of XML I’ve marked B.

Step 3 is to copy that <zone … id=”X” …>…</zone>  element of XML (in my case X=1, but your case will likely differ). I’m going to paste that block back in twice (as I want two more text boxes) and I’m going to paste at the bottom of the zones section, just before the </zones> closing tag, as you can see in the next screenshot:

copy-paste-03



What you’ll see I’ve also done is

  • updated the id=”…” for each of the copies I’ve pasted in to be the next highest id number based on the preceding zones.
  • updated the x=”…” and y=”…” coordinate values appropriately (I made my life easy here by having a 2×2 grid where I’d already added elements to two spots, so I can just copy the appropriate x and y value from preceding zones, and I didn’t need to edit width and height). Don’t be phased by the x, y, width and height values not looking like the corresponding pixel values in Tableau Desktop. You can always grab your calculator and work out what you need from other values, or just offset enough from other zones that you can subsequently fix it up in Tableau.
  • finally I updated the text in the formatted-text > run elements. You don’t have to do this here in the text editor though, as you’ll be able to edit it in Tableau Desktop too.

Cool. Save those changes, close your text editor and …

Step 4: Reopen the workbook in Tableau Desktop. You should see that the pasted text boxes show up with pretty much the same styling:

copy-paste-04



There is one slight problem though - the text alignment isn’t the same. We can fix this!

Step 5 in this guide is to close Tableau Desktop, reopen the .twb in a text editor and add a bit more XML. Obviously if or when you are doing this for real you’ll do step 5 at the same time as step 3 above. We need to copy and paste the text- and vertical-align format styles too as illustrated here:

copy-paste-05



You’ll see that I’ve had to derive the relevant id=”dash-text_X” value. The X matches the id chosen for the previously pasted in zones. Save your changes again.

Step 6 is to reopen in Tableau Desktop and you should see correctly aligned text:

copy-paste-06



There you go. Copy and pasting text boxes in Tableau!

Yeh, but…

I went on to try copy and pasting a non-text box zone. I copied a chart zone and edited the id, and changed the name to another unused chart from my workbook. When I reopened the workbook in Tableau Desktop I got an error. The error told me to contact support. I’m pretty sure Tableau Support don’t want to hear from me after I’ve hacked the underling XML. And I’m pretty sure I know what one of their first questions would be; “do you have a backup you can revert to?”!

I’ve used this approach to copying and pasting text boxes for a couple of Makeover Monday submissions, but not on work projects. Nevertheless I believe I’ve learnt a bit from the underlying XML behind my workbooks, and it makes me think that if the idea referenced above is up voted enough it wouldn’t be a big stretch for the development team to iterate some potentially very well received functionality.

PS. if you want know more about XML, then check out this resource.