[MUSIC] Welcome back, on A Primer on Business Experimentation. We are going to explore the concept of business experimentation in a little more detail. Recall that what we did just prior to this was basically motivate the need for business experimentation. Get some insight into the type of problems and the type of data it can handle, and solve for it. Recall the anatomy of analytics. Basically, this comes from session one, if you recall, and we basically saw real world problems involving real world questions and looking for real world answers. Basically, the bridge between real world questions and real world answers could be of two types. The first approach was the direct one and that was the one we just motivated just now, right? So experiments directly in some sense lead to real world answers. The other one, and we'll come to this later in the session, is to actually model it, and use the analytics route. All right, so let's get there. First thing first. What is the formal definition of an experiment? Well, the act in some sense of conducting a controlled test or investigation. And the keyword there being controlled. There is a control. That is a controlled condition, a controlled group, a controlled sample. Why are formal experiments called causal research? Well, causal basically comes from the cause in because, cause and effect. Experiments are called causal research, because they and they alone, in some sense, allow us to answer the question why. Most other techniques are associative, correlational in nature. Experiments alone help us answer the question why. The main components of an experiment? Well, there's treatment. There's control. There's sample selection. There's outcomes measurement and we will see all of this in what will follow just now. The experimentation method in a nutshell is basically this. We measure outcomes corresponding to different treatments and infer that the treatment difference caused the outcomes difference vis-a-vis a control, a control group where the treatment was not applied. All of this was abstract, right? To get a little more concrete, let's actually take a real world example. Let's go to reading one. And after this reading, we'll basically go to what the insights from the reading are, what the learnings are from there. Well, notes on reading one. The SOP, Standard Operating Procedure of experimentation techniques for business is this step one. Existence of a confirmatory or a causal RO. Remember what an RO was? It comes from module one. A research objective. Unless there is a research objective defined, do not jump into experimentation. We need to know what it is that we are testing for. And the causal RO is typically framed as a business hypothesis. Two, step two, clarity on what is treatment and what the outcomes of interest are. Unless we know what the treatment is, we will not be able to measure anything successfully. Measurement at scaling both come in at this stage. Again, concepts borrowed from what we covered in session one. Step three, formation of a treatment group and control groups. And this is important. I need to have a treatment and a control group separately. The treatment group is exposed to the treatment. The control group is not exposed to the treatment. The difference in outcomes, we will infer is caused by the treatment. Step four, the treatment and control group ideally should be identical, but failing that should be made as comparable, as equivalent as possible. And we saw a bunch of techniques in the reading that in some sense tried to make this happen. So we saw clustering, something that we will see in session three. Statistical matching, randomization and so on. And finally, the use of classical statistics to infer the significance of the result. Basically, it answers the question that what I see as a result, is it something that happened to happen randomly or is it systematic? So with what confidence can we say that the effect is what it is? Which brings me in some sense, experimentation for marketing research and introduction. Experimentation should be considered whenever you want to compare a small number of alternatives. Any guesses on why we're looking at only a small number of alternatives? Because as the number of alternatives gets larger, the number of cross connection of factorial design, the number of things to control for, just increases exponentially. Six, some say, is a practical maximum for the number of alternative you should in some sense handle. After that, they'll all start to get noisy. Two, an experiment is only as good as its degree of control, how well the treatment and the control have been defined and measured? In some sense, becomes critical here. Has it been isolated and administered correctly? Were the respondent groups equivalent? If they are not, then the results of the experiment are called into question. There are two main types of experiments, a field experiment and a lab experiment, and each has their pros and cons, right? So the con of a field experiment is defined as not everything can be controlled, like a controlled environment that you have in a lab, and the fact that it is expensive. The test to market is a field experiment and it is very expensive. Which brings us in some sense to some contemporary questions in business experimentation. Think about it. A firm is doing email campaigns, okay? It wants to know the answers to these everyday questions that an email campaigner comes across in day to day life. What day of the week gets a better response, right, open rates? Does a subject line with an incentive or a teaser work better? Does including your company name in the subject line increase engagement? Should you in some sense include your name or your firm name in the email header? Does the time of the day matter? Are mornings better than afternoons or vice versa? A subscriber is more likely to click on a linked image or a linked text. And all of these in some sense are everyday questions that an email campaign, or email marketer is going to face. Question to you. Suppose you are a consultant to this email campaign manager. How would you find these answers? Suggest a systematic approach, outline it, take a couple of minutes, write it down. Which brings me. All right, to in some sense an attempt to answer those questions that we saw on the previous slide. And one way, in some sense, we could go about answering this. Consider the first question. Subject line, what should we have in the subject line? Suppose you have two choices, you could have more. Suppose you have two choices, you have a long list of people you're going to email. You take a small sample of those people, split it into two, that sample into two, and then try Subject Line A on one group and Subject Line B on the other. If they are randomized, the hope is in some sense, they are equally willing. If that happens and one group actually responds better to their subject header, automatically, we can infer that that subject header is better. And for the rest of the list, we are going to use the better one. Okay, there is one instance in which both lab and field testing are the same thing. And basically, what your platforms are and what is called A/B testing. So think of virtual platforms like a Facebook or a Google, or a Yahoo. Well, for them, virtual platforms, A/B testing is the same as actually business experimentation, right? What configurations, what elements, what buttons to place where, what outcomes? All of them in some sense. Yep, and with what particular frequency all of them become important. Take a look at this. A/B testing means I want to split the sample into A and B. I'm going to test treatment A on one and treatment B on the other. And basically, look at which works better and go with that. The data are speaking, there is no complex theory involved. Let the data speak. Let the real world speak, right? So that's basically the answer. Here's an example, A/B testing in real life. The year is 2012, the venue is the US of A and the event is the US Presidential Election when Barack Obama was competing with a Republican opponent, a guy called Mitt Romney. Now Obama's campaign had an 18-person team that tested over 10,000 email versions, okay? For instance, just for subject header there were 18 variations, and they were trying to raise funds. So a fund raising campaign from the crowd, right? And they tried different subject headers. The most effective header was, I will be outspent, that basically raised $2.67 million from ordinary people, right, who contributed in small amounts to the campaign. The one thing that polls got right, was the least effective. It only got about $400,000, okay? Now the results of this data-driven fundraising campaign were impressive., why? Because in 2012, Obama went on to raise $1.123 billion in total, out of which more than half, 690 million actually came from about 4.5 million small donors, all contributing small amounts, okay? And this was 4x the amount that Romney was able to raise. So yes, maybe design works and there are real outcomes in monetary terms out there in the real world. [MUSIC]
