The Complete Guide to Subject Line Testing

/ Zach Heller
inbox.jpg

Subject line testing is a crucial part of any email marketer’s role. It is singularly one of the simplest and most effective ways to improve the performance of an email marketing campaign.

But saying it is simple is one thing. So today I wanted to actually take the time to lay out the do’s and don’ts of subject line testing so that every brand, organization and marketing team can succeed in their quest to find better subject lines.

What is Subject Line Testing

Essentially, subject line testing is the process by which you seek to improve the open rate on your emails by delivering the same email to different audiences – one of which sees one subject line, and the second group sees the other. In this way, subject line testing is a very specific example of an A/B Test.

The reason that brands of all types run subject line tests is that it is one of the best ways to improve the overall performance of an email marketing campaign. Because the subject line is one of the first considerations for an email recipient – one of the first things they see along with the sender once an email lands in their inbox – it is one of the most important elements of any email.

Often, a subject line alone will mean the difference between a recipient choosing to open the email or not. So by testing multiple subject lines, we aim to improve the open rates over time.

How to Set Up a Subject Line Test

There are a number of different ways to set up a subject line test:

  1. You can test the entire list. If you are testing an entire list, you simply send 50% of them subject line A and the other 50% subject line B. These results will help you plan for future emails.

  2. You can test just part of a list. You might choose to only test part of the list, so that you can use the results of the test to send the winning subject line to the rest of the names on the list. For example, you might take 20% of your list and split it into two groups (10% each) for the test. Then wait 24 hours to determine the winner before sending the remaining 80%.

  3. You can test programmatic emails. If you have an automated email campaign, one that triggers based on user actions and often includes more than one email in a sequence, you can build tests into your program. Send half of your users down one path and half down another, and run subject line tests at each step in the process. However, here you must remember that the sequence matters, meaning the results of one test at the beginning could impact results of emails that come later.

How to Judge the Results of a Subject Line Test

Judging the results of a subject line test should be quite easy. Remember from the information above that the goal of any subject line test is to improve the open rate of that email. And so it is the open rate that we must measure.

Compare the open rate of subject line A against the open rate of subject line B, and you should get your answer. The email with the higher open rate is the winner.

A few additional things to consider:

  • Allow enough time between the email send and the results analysis. Not everyone will open the email right away. A good rule of thumb is to wait at least 24 hours to declare a winner.

  • Statistical significance matters in any A/B test. A result is considered statistically significant if the confidence level is at least 90-95%, meaning you can confidently conclude that the subject line was responsible for the difference in open rates (rather than chance). This is a great free tool to measure statistical significance.

  • It can be tempting to look at metrics other than the open rate, such as the click through rate. But the subject line will have less impact there. If the open rate on one version is significantly higher, but the click through rate on that version is significantly lower, you might consider what other factors are leading to those results.

What Types of Subject Lines Should You Test

As a general rule, you should test all of your subject lines. There is no surefire way to know which subject lines will perform best for an individual email. And testing is the only way to determine that with any certainty.

But just like anything else that you do, some measure of priority must be determined. Which emails are the most important? How would an improvement in the open rate of each email impact the business at large?

Once you have determined which emails to test, there are some best practices you can aim to incorporate into your new subject lines:

  • Personalization – include the person’s name or other details in the subject line

  • Urgency – convey a sense of urgency around offers

  • Questions

  • Punctuation

  • Less is More – limit the length of your subject line

  • Capitalization – try different ways of incorporating capitalization

  • Active Words

  • Positive language vs. negative language

You can check out this tool from SubjectLine.com, which calculates a score based on a very specific formula for subject line effectiveness. But remember that that is just a guide and just because a subject line does not score well doesn’t mean that it won’t perform well with your lists.

Why You Should Send Your Next Promotional Email at Midnight

/ Zach Heller
stopwatch.jpg

When a politician changes their position on a policy question, we call them a flip flopper. And we don’t trust them.

But changing your position on an issue based on new evidence is a sign of progress and wisdom. And not just for politicians, for all the rest of us.

For a long time, when it came to email marketing, I was a firm believer in wanting to land in an empty inbox. I believed, and still do to a large extent, than one of the best ways to stand out and grab attention is to avoid the crowded inbox completely.

This led me to guide you all to send on days – and during times of day – that may not be as common for other companies to send their promotional emails. The idea was that if you avoided the “busy times”, you could show up at the top of your prospect’s inbox, the lone unread email waiting to be opened and read.

And when I got some advice that suggested I send emails in the middle of the night instead, I brushed it off for exactly the same reason…at first.

I thought, if I send an email in the middle of the night, it will show up in the flood of new unread emails all screaming for attention when people check their inbox in the morning. As one of many, my email is less likely to be read or carefully considered – more likely to be deleted in a sweeping purge of unwanted pre-caffeine stress.

But when I saw proof of another company who had made this strategy work, I decided the time had come to test it for myself. And lo and behold-

The Midnight Send

I took a popular promotional email and sent 50% of the list at the normal time, which was 11am on Tuesday. The other 50%, the test group, I sent at midnight the night before.

And when I checked the results a few days later, here is what they looked like.

  • Control Group: 23.4% opened, 4.2% clicked.
  • Test Group: 36.7% opened, 7.7% clicked.

That’s a whopping 57% improvement in opens and 83% more clicks. And it was no fluke. I tested it again the following week and got similar results.

The Theory

The hypothesis was this – for some subset of consumers, the morning is when we are most likely to check our emails. And so by ensuring that your email is waiting in the inbox first thing in the morning, you are ensuring it gets the attention that it would not get at any other time of day, when people are too busy to check email with any regularity.

By sending overnight, you are adapting to your customer’s behavior, rather than trying to get them to adjust to yours.

So my message to all the email marketers out there is simple. If you have not done so to this point, test your next promotional email at midnight and see whether or not you see a similar lift in performance.