In this CRO Live Hour Podcast episode, hosts Khalid & Simbar talk about the Conversion Framework. They mainly focus on three questions:
QUESTION — Have a question about e-commerce Conversion Optimization, A/B Testing, User Research, Marketing, or Anything Else? Post in the comments section of this video!
Visit Our Website, And Learn More About Our Services:
invespcro.com
Follow Invesp on Social Media:
Linkedin
Facebook
Twitter/X
Read more about conversion rate optimization on our blog.
In this CRO Live Hour Podcast episode, hosts Khalid & Simbar talk about the Conversion Framework. They mainly focus on three questions:
QUESTION — Have a question about e-commerce Conversion Optimization, A/B Testing, User Research, Marketing, or Anything Else? Post in the comments section of this video!
Visit Our Website, And Learn More About Our Services:
invespcro.com
Follow Invesp on Social Media:
Linkedin
Facebook
Twitter/X
Read more about conversion rate optimization on our blog.
[00:00:00] Simbar: You also mentioned that most of the experiments That we've been launching for a certain client are focused on engagement, Can you give us like an example of those engagement experiments?
[00:00:12] Khalid: Yeah, so engagement experiments sometimes uses UGC, user generated content, correct, to create engagement. We also, and this is interesting because we have Visitor orientation. So you orient the visitor. Here you are. Here's what I want you to do.
[00:00:26] That's under-engaging the visitor, correct? Having the visitor engage a little bit more with different elements on the website. Those sometimes are very useful. Imagine, I'll go back to the example of the business broker. I'm a doctor. My clinic is generating a couple million dollars in annual revenue.
[00:00:44] That's my run over. And now I am thinking it's time to retire. I'd like to know,
[00:00:49] Intro: My name is Khaled Saleh, and I love to talk about conversion rate optimization. I'm Simba, and I love asking questions about conversion optimization. This is CRO Live Hour, a [00:01:00] show all about A B testing, experimentation, and conversion rate optimization. Each episode, Khaled amazes me, answering some of the difficult CRO questions, dropping insights like it's no big deal.
[00:01:10] While pretty much every episode, we'll take on a new set of conversion rate optimization questions that are not easy. We will talk strategies, process and we will talk tactics. Simba will be bringing all the questions. Oh man, I bring tough questions like, do A B testing results fade over time? How do you go from low to high testing velocity?
[00:01:29] How do you measure the success of a conversion funnel? And how do you align your CRO program with a growth strategy? Yeah Khalid, these are very, very tough questions. Yes, they are, but we always answer them here. And if you love conversion optimization like we do, and certainly like Simba does, subscribe to the CRO Live Hour podcast today, wherever you listen to your podcasts.
[00:01:52] Khalid: here we are, Simba.
[00:01:54] How is your week? I know it's been crazy busy. We wrapped up. February. So 66 percent [00:02:00] of Q1 is done. How is life? How is the first quarter coming along?
[00:02:04] SImbar: I think life is good. Of course, there are some I wouldn't say disappointments, but certain like lessons that you learn along the way.
[00:02:12] There are some predictions that we made. Some of them, like they are working, some are not. And we're also like trying like some new things. That's the beauty of all this, like trying new things. You get to learn a and you try to improve on what works and what doesn't work. And one thing that I've also learned or I realized about marketing is You really can't use last year's plan this year because a whole lot of things changed.
[00:02:37] But you can borrow some of the activities that have been working for you. it's a new year. You don't know what will happen.
[00:02:43] Khalid: Okay. I have to ask you as you mentioned that, what worked last year that didn't work this year? Every time you throw a statement, somebody expect me to dig a little bit deeper.
[00:02:52] Putting you on the spot, what do you think worked really well last year that's maybe not working so well this year? Maybe it's a little bit less [00:03:00] effective. And what do you think is working in 2024? It's funny because I took this post and I made it. Hey, how do you improve a website conversion rate in 2024?
[00:03:08] But we'll talk about marketing first and then we'll talk about conversion.
[00:03:11] SImbar: yeah, you really had to put me on this boat and that one. Now I have to think.
[00:03:15] Khalid: I just want you to appreciate something though. I could have just said the question and stopped, but guess what I was doing? I was talking and talking and talking. I didn't add a whole lot to what I was saying.
[00:03:26] I was just giving you a chance to think, such as right now. I'm giving you a chance to think, I'm hoping that you'll be like, okay, here's what worked. But tell us what worked, what didn't work, what doesn't work anymore.
[00:03:36] SImbar: I think like one of the things that really works like throughout the year, or maybe I can say like ever since like I joined Inverse, you It's the personal branding thing.
[00:03:46] Of course, most of the people, or if not, like all of the clients that really reach out to us, they reach out to us because they want to work with either you or IAD. So it's always like the personal branding that is one. So as a [00:04:00] marketing team, one of the things that we have to do is to amplify like the personal brand, that's working.
[00:04:05] uh, Either your brand or your ads brand. So I would say that's one of the major things that works. How do you then amplify two brands? There are a couple of ways of doing that. we do like a a lot of videos at Invesp. We also do LinkedIn, but not as much as we used to do.
[00:04:23] So I would say like in terms of what really works, that does work. So we also like, went into focusing LinkedIn ads, LinkedIn ads, they're interesting because we went into LinkedIn ads. I think that's like a something, a mistake that we did without really, focusing on our ideal ICP, because like we just assume that a lot of people. that join a prospect for invest there on LinkedIn. But some of course they're just on LinkedIn, but they're not that super active on LinkedIn. Whenever like you go and create like your audience on LinkedIn preparing for your [00:05:00] ads.
[00:05:00] LinkedIn will always show you that those people are there, even if like they are not like really active. So that was also like one of the major lessons I learned. And also one of the things that I've been doing. Especially like, this week I was going through all the leads that we got through Google ads and trying to see how good are they in terms of the quality of the leads.
[00:05:23] I was also looking at ads that we are using on Google ads and trying to see. If there's similarity or how are we differentiating ourself from the competitors? Because most of the keywords that we are going for Google as the turn of companies that also via go for the same, keywords So I was just like trying to look at like how can how best can we differentiate ourselves?
[00:05:44] So yeah, those are some of the things that we've been working on about Google ads. I would say like it works sometimes it doesn't work sometimes. So I think also the issue of seasonality at some point might come into play because there are [00:06:00] certain days that I noticed that our Google ads like perform best, like especially like Mondays is our best day.
[00:06:05] Wednesdays is the second one. Then I was so surprised to see that Friday, Saturday, Sunday are no days for us. So what does that mean for us? Should we turn off or pause the ads on those days and just try to make sure that like we maximize our budget on those kind of days.
[00:06:21] So those are some of the things that I was just like trying to look at to see. What works and what
[00:06:26] Khalid: doesn't work. I like that. I like that. There's a lot of lessons there. A lot of lessons that cost a lot of money, by the way, for those are listening. I know we've just added a new field to all our forms, which is the attribution to understand where somebody had heard about us first.
[00:06:40] So it will be interesting. This will be an interesting conversation in three months. We would have had enough data. I truly wonder how that will impact. Our marketing, sort of sad. I don't know sad. I'm being the conscious decision. I'm like, oh, we should not have that field and now I look back at the decision 2024. I'm like, so why didn't we have that [00:07:00] decision? I've always made fun of people who have had that field. I'm like, you can figure it out. No, you cannot. And We've had the opportunity to know that. But now we're doing whatever it takes to figure that out.
[00:07:10] And then we'll adjust our plans accordingly.
[00:07:12] SImbar: also think that's really the funny thing, because sometimes I might say that maybe our Google ads are not working, right? Like, when I look at that funnel of Google ads, I see like, there are no leads coming in. But sometimes when we post it, It might also affect the organic or other finials, so it's so interesting.
[00:07:31] Yeah,
[00:07:31] Khalid: Yeah. it's not solved until now, which is the problem of attribution in in marketing. The reality of it is really there's not a good solution out there. Everybody struggles with that. But okay, so that is marketing. Let's talk about, I said, okay, and then that's how I started the post today by asking a question and saying how do you identify conversion problems on your website in 2024?
[00:07:56] I just changed the title a little bit, but we're still discussing the same topic. [00:08:00] Because I think everybody's trying to figure out. CACs going up, ROAS is going down, our CPAs are going up. How do we actually do better when it comes to conversions? I was talking to somebody and he's like, Oh, I'd like to grow my revenue by 70, 80%.
[00:08:16] I'm like, okay. My next question to him is What are you doing right now? He said, Oh, we're doing about 20 million. We'd like to hire you guys. So you can come in and help us grow by 70 percent and 80 percent in 2024. And I told him we're not a good fit. I know we do absolutely amazing work, but that's 70, 80 percent growth.
[00:08:36] Anybody who wants to take credit for that is probably It's not telling you the truth. Yeah, it's part of it is experimentation. CRO part of it, the CEO, PPC, too many things working together and getting lucky at the same time. And you have the right product market fits and you have amazing product that people have, and then you grow by that much.
[00:08:52] Now, remember this is for a company that's doing 20 million. This would have been a different conversation if you had told me, Oh, we're doing half a million. Okay. I can see [00:09:00] that, but the 20 million to grow by 80%, that's another 60 million. That's doubling the size of your business and a 20 million just logistics around that and whatnot.
[00:09:09] But anyways, let's talk about how you improve your site conversion rate in 2024.
[00:09:13] SImbar: Yep. That's interesting. As you were saying also like about that question, I was like thinking of, is there a difference? between how the conversion rate used to be improved back in 2006 when Invesp was started.
[00:09:25] And right now in 2024, then like, in my mind, I was like, Oh, websites were very, bad back then. So today we're talking about the conversion framework. Yes. This is especially the conversion framework we're going to focus on is the one that you developed, I think, in 2007, if I'm not mistaken.
[00:09:44] Khalid: Yeah, 2007, that's when we started using it.
[00:09:47] SImbar: So can you tell us more? What is a conversion framework? What is it? why is it important
[00:09:51] Khalid: to have one? Yeah, start by the story of how the conversion framework happened. So when we first started in experimentation [00:10:00] now remember, this is a time where there even the name CRO did not exist, even it was AB testing.
[00:10:05] We would not even use the word experimentation by the way, or AB experiments. We take a look at the website, we take a look at the page and we say, Oh, we should change this. We should change that. And it was purely relying. On the fact that we surfed lots of web and we've studied usability and that's that.
[00:10:21] Now we were lucky because yeah, most websites were really bad. And we would make changes and oh gosh, like the most of our experiments would produce amazing results. I still remember one of the first projects we worked on and we had a co sponsor from Microsoft. Didn't really quite believe in A B testing at all, but it's like, okay he got stuck with the project, so he's going to do whatever it takes.
[00:10:42] We launched the first experiments on a bunch of landing pages that they're running and conversion rate literally increased by 238%. I still remember the number because I still remember the email that he sent like a week later. He's like, Oh, I'm a believer now because he used to get about a thousand leads and it jumped up to about 2400 leads.
[00:10:59] Per [00:11:00] month, you cannot debate that when you have a single landing page driving traffic to it, and you see, it's not like, you know, from 1000 to 1050. It's, from 1000 to 2300, at least leads, things are great. But then you start thinking to yourself, and then you end up with this one client, who's also not a believer.
[00:11:18] And again, this is when the industry was new. And he's like, Oh, so just randomly guessing different things. It's like, you're just looking at a page and you're saying, Oh, we should change this and we should change that. And the reality is that's what we were doing. We were just guessing different things.
[00:11:33] It was a slap in the face when he said it. He's no, of course we are the experts. We know what we're doing. But when we, me and Ayad sat back and we said, you know what? There is truth to that. a little annoying that we heard that. But we probably need to bring a little bit more structure.
[00:11:48] Eventually we want to add more people to the team. And guess what? When you add another person to the team, either hire somebody who has the same level of experience as us, or you say, you know what, here is the process that [00:12:00] you follow when you're evaluating a page, when you're evaluating a website.
[00:12:04] So the initial thoughts, and it's funny how the conversion framework evolved, is just, oh, let's list everything that we can think of, all the problems that we found. And we've done a ton of projects at that point in time. And that was the first step at attempting to create the conversion framework.
[00:12:18] But then we stopped and we said, okay we're just constantly thinking of new things and we need a better way to categorize those. So again, we stopped and we said okay, if a person, if a human lands on a website, What does that person need to see in order for them to convert? Let's say that person that comes to the website is in one of three states.
[00:12:39] One is, Oh gosh, I'm here by mistake. I need to leave right away. Oh, I'm shopping and browsing. I really don't feel the need right away. Or the third category. No, I am actually looking to buy this product and I have no issue like not paying for this product. So different people come to the website at different stages and then we start thinking about, Okay what do you need [00:13:00] to present to those people?
[00:13:01] How do you structure your website? How do you structure the visitor flow? How do you structure the design, the copy, all the elements on the website? So they're all in harmony so they can present that experience to the user. And we start thinking about the brick and mortar. experience that you have.
[00:13:16] When you say, you know what? I am going to go and you're going to find this funny. Oh, I want to go buy a DVD player. That's what we were thinking back then. No one even try and tell that to my kids and they're like, Oh, what's the DVD? I'm like, Oh, leave me alone. But You say, Oh, I want to buy a DVD player.
[00:13:30] So there's a need. And then you say, Oh, I'm going to buy it from this store. And then you walk into the store and then the salesperson is meeting you and asking you some questions and guiding you to select the right DVD player. Okay. So how would a website do that? That's when we start thinking about the conversion framework, it's different elements and how they impact the visitor.
[00:13:50] And. Over the years, we've continued to evolve the conversion framework. The seven principles of the conversion framework, the seven guiding principles, what we call the main elements, [00:14:00] are still there, but they're sub elements, correct? So each principle have what we call sub elements and we have what we call third level elements.
[00:14:07] Those continue to evolve. And it's funny because that is one area where IAT is what we call the gatekeeper. For us, she is the one who is responsible for maintaining the list, not the first element and the second element, because those are very well defined at this point, but it's the third element that we have in those kind of a way to drill down as we're evaluating a problem on a website and every now and then we'll have a CRO who's looking at the website, they identify a problem, they say, okay, so which element of the conversion framework this addresses and they might not find it there.
[00:14:40] So I'll have a discussion with I had to decide. Oh, it's actually probably to classify it in a different way. Or should we add a new element or a third element to the conversion framework? That's probably a little too abstract on how I gave. But how the process works.
[00:14:52] SImbar: Yeah, that makes sense.
[00:14:53] Let's get like into the seven principles of the conversion framework. Can you give us an idea what those [00:15:00] are?
[00:15:00] Khalid: Yeah, the seven principles number one is trust and confidence. Very simple, very straightforward. If I don't trust you, I am not gonna buy from you.
[00:15:08] Number two is FUDs, fears, uncertainties, and doubts. We all carry them as we transact. The minute you ask somebody to give something out, whether it's time, whether it's money, whether it's effort. They have some funds, fears, uncertainties, and doubts that will stop them from spending, especially on the web, their hard earned money, and sometimes even giving away their email.
[00:15:31] So you need to increase trust. You need to reduce FUDs. Two other elements that are critical. One is incentives. How do you incentivize somebody who wants it? How do you incentivize somebody who just landed on your website to act right away, as opposed to saying, Oh, I'm just going to leave this window open or this tab open on my browser and I'll come back to it later.
[00:15:52] Or maybe I'm just going to bookmark it and I'm going to look back at it at a later time. So I want to incentivize you to act right away. Number four [00:16:00] is engagement. How do you engage the visitors? So they're spending more time. They're like, Oh, yeah, all walk into a store. And. We might get engaged as we're going through the aisles.
[00:16:11] This is physical store, correct? Or we might walk into the store and we're just nonchalant, could not care less, I want to leave right away. I've decided about a couple months ago that I'm no longer shopping at Walmart. love Walmart, but I've decided no more shopping at Walmart. So I've been trying to find replacements.
[00:16:27] I go around the area, and I find this store is big. Not as big as Walmart, but it has a ton of stuff. Mariano's. So I'm like, oh, let me just go and walk in. Literally over the weekend, I'm gonna go there. It's amazing. I walked aisle, after aisle, and I just was not interested in what they have. There's a much smaller store, and I'm in the Chicago area.
[00:16:46] A much smaller store. Called Pete's, but somehow the experience that Pete's gets they get you fresh fruits, fresh vegetables. they have literally, because we live in a multi ethnic area, so there's lots of Polish guys, lots of [00:17:00] Arab guys, Indian, so they have aisles that are just dedicated to each one of those countries.
[00:17:05] And I'm like, oh, this is so cool. I just walk through one of those aisles. That appealed a lot to me. For me as a customer, that just worked really well. So engagement is a very important element and it's funny. Simba, if you had asked me. Five or ten years ago about engagement, I'm like, eh, it's not an element that we spend a lot of time on.
[00:17:22] Nowadays, I was looking at data from our testing program in 2023 across all the different clients. You're talking about close to 2, 000 experiments. Engagement eats up about 40 percent of the experiments that we run of all the seven different elements of the conversion framework. So engagement matters and matters a lot, which is fascinating because that was not the case.
[00:17:41] Those are the four main elements and then we have what we call, I like to call them kind of supporting elements. So we have the sale complexity. Not every sale is, Oh, I'm coming. I'm giving you my credit card and I'm just walking away. Correct? Imagine. We work with one client, [00:18:00] they help doctors, they're a business broker, they help doctors sell their clinics to corporate buyers.
[00:18:06] A transaction value is about five million dollars, five on average. And not only it's a huge dollar value for the transaction, but it's a life changing transaction, correct? Because when you're selling your practice, for the most part, you're retiring. Now, I tell you, over the years, I've worked with over a thousand clients.
[00:18:25] This particular client is the one client because one of the things that we do is we tell them, Hey, can we interview some of your own clients who have worked with you? Some of those doctors? This is the one client that I've interviewed probably seven different doctors who have worked with them. I personally interviewed them and They're like, oh my god, they are life changing I've never had Raving fans, correct? I mean you have people who are happy with a product or a service, but this particular business broker you just hear it and I'm like guys you don't have clients. You have raving fans They're just in love with you and the way you do service.
[00:18:56] So sales complexity is really [00:19:00] important the buying stage is very important. Not everybody who comes is ready to buy right now. Somebody is in the research phase. Somebody is in the maybe compare alternative. Somebody is in the action stage. And then the final thing that we have, which we used to rely on a lot, and I think it's a little bit less important, is the personas.
[00:19:21] Different people buy differently. We still use personas, but I think it's been replaced a lot with the jobs to be done. framework thinking about the social, about the functional, social, emotional elements of making a purchase. So those are the seven guiding principles that we have when it comes to the converter framework.
[00:19:39] That's
[00:19:39] SImbar: cool. I just have a couple of questions on most of the principles. I think like trust is straightforward, everyone like really knows trust is important. Then we touched on it sometime. I want to talk about the facts, which is the fears, uncertainties, and doubts.
[00:19:54] How do you spot those on a website? And when you see like [00:20:00] one, do you always have to run experiments? In my mind, I'm thinking that some of them might be usability issues need to be fixed
[00:20:07] Khalid: right away. Let me start by the end of the question, and then I'll go back to FUDs.
[00:20:12] So whenever we're evaluating a particular page on a website, We evaluate those general principles, but remember that you also have the 10 NNG principles, the 10 usability principles. It's been around for, what, now, 10, 30 years, probably close to 30 years. Those, by the way, NNG principles. come and fall under different elements within the conversion framework.
[00:20:35] For example, showing system statuses, that's under FUDs because if I'm submitting something and I don't know matching real life systems, So there's 10 principles and we find that they fall under the conversion framework elements. that's number one.
[00:20:48] Number two, there are certain things that you can right away assess in terms we For each one of those elements. So if you think about fuds fuds for us breaks down to about 50 different [00:21:00] elements on a website and each one of those 50 elements has probably three, four different things that we have.
[00:21:04] And sometimes it's very obvious, correct? I want to know the shipping date. I want to know the shipping price. It's very obvious, but sometimes it deals, for example, with anchoring the price, anchoring expectations, anchor, There's things around that. Some of them are psychology principles.
[00:21:16] Some of them are usability principles that we need to identify. And as we're going through the page, we're looking at every element. And we're asking ourselves, do we see a usability issue over here? Do we see an issue where people might get confused? So that's one thing that we do in our evaluation. At the same time, there are two things that happen that are very powerful in identifying FUDs, jobs to be done, interviews, those lengthy interviews that we go through, understanding the customer journey and the concerns that they have before they make the purchase decision.
[00:21:48] What were those concerns? Who did they talk to? How did they those concerns? What assurances were they looking for? to make sure that they're making the right decision. I was looking at something called Pepdex. And if you've [00:22:00] seen it, I'm going to share it with you because I just bought it.
[00:22:01] It's a way to help you with storytelling. And it's expensive, it's like 200 bucks. Okay, it's not that expensive, but it's expensive nonetheless, I'm like, okay is it useful or not? That was the biggest concern that I have. but to me, I just don't want to buy something that I will find not very useful, that was the big question.
[00:22:17] And obviously they've understood that because even in their ads on their Facebook, I visited their site. So they're hammer me with their retargeting ads on Facebook. They're showing a whole bunch of customers who have used them, who found them to be very useful is that I did not trust. Any of those ads, by the way, which is very funny, guess what I did?
[00:22:36] I went on Reddit, I searched for them, I found these big threads, one thread had almost 200 comments about it, read all the comments. So I'm like, okay, that gave me some confidence, and then they had something else, which I know I'm probably not gonna use, but nonetheless, it just gave me a little bit more confidence and dealt with my fears, uncertainties, and doubts whether this is useful or not.
[00:22:56] They said, if you don't get 10x the return from [00:23:00] those PEP cards as much as you spent, return them within 365 days. Powerful. I bought products that did not work and had guarantees and I just like never actually use the guarantee or the warranty. But the fact that you are so sure that you give such a guarantee, now it might be a very smart marketer who says, listen, statistically less than one and a half percent would return a product.
[00:23:25] So our 2 percent will return a product after they buy it. So why don't we do that and just mimic, send the message that we are. Behind our product. So much. So again, the customer journey helps us with that. And then the third element that helps us in identifying funds on the website, sorry, is by asking people.
[00:23:43] we like to launch on site polling where we say Hey quick question, what's stopping you from buying today? And when we collect 300, 400, 500, a thousand answers and then we start analyzing them. And then we say, okay, now which ones are reoccurring? Many people mentioned them. How do we deal with [00:24:00] that?
[00:24:00] those are some of the ways that we identify FUDs. going to finish with this. I shared something on LinkedIn where we did this for one of the companies that we're working with. And we collected almost 400 answers. And the most common answer was price, too much, it's too expensive, it's too expensive.
[00:24:15] And then somebody, the reason I shared the post because somebody says, what's stopping you from buying? He's my wife, and I thought it was just very funny. And somebody else, my bank account. And I'm like, okay, guys, don't know how to help somebody with his wife. But. If 80 percent of our visitors are saying they're ready to buy, but they cannot buy because of price, we need to justify for them the price and why they should buy from us because of the product quality and whatnot.
[00:24:36] So again, we've identified an issue and then we said, there's different ways that we need to address that. that's how we deal with funds.
[00:24:42] SImbar: Yeah. You said a lot of interesting things there. And I also like what you said the last statement that you said that when Most of the visitors like complain about the price.
[00:24:51] You don't have to offer discount, lower it, but you just have to justify. Because most of the people, they might think that, ah, maybe we just need [00:25:00] to decrease the price because it's too high. So that's interesting. You also mentioned that most of the experiments That we've been launching for a certain client are focused on engagement, Can you give us like an example of those engagement experiments?
[00:25:16] Khalid: Yeah, so engagement experiments sometimes uses UGC, user generated content, correct, to create engagement. We also, and this is interesting because we have Visitor orientation. So you orient the visitor. Here you are. Here's what I want you to do.
[00:25:30] That's under engaging the visitor, correct? Having the visitor engage a little bit more with different elements on the website. Those sometimes are very useful. Imagine, I'll go back to the example of the business broker. I'm a doctor. My clinic is generating a couple million dollars in annual revenue.
[00:25:48] That's my run over. And now I am thinking it's time to retire. I'd like to know, I would like to know how much money I can get for my business. Am I going to be getting 2. 5 million? [00:26:00] Am I getting 5 million, correct? That's a question. So having a calculator where people interact and it gets an initial estimated value, but guess what?
[00:26:07] This is just an initial estimated value. If you need us to give you a more concrete value, then we need more information from you. So again, It's almost you know, the principle of reciprocity, correct? I'm going to give you something of value, I'm going to ask you for something else, and there's an exchange of value that is happening over there.
[00:26:24] I'm just looking at data here. I'm going to actually zoom in, and this is interesting, by the way when we look at this, let's see Here we go. Engagements and you know what? I'm going to actually filter. So I'm actually looking at a client data. How about that? And it's just fascinating when you do that.
[00:26:41] So we're just going to clear all of it and let's take a look for this particular client. 35 experiments ran in the last six months 35, I would say 33%. 12 were ran around engagements. Again, this is one of the seven elements. So statistically [00:27:00] should be around 12%, not 33%.
[00:27:01] So again, that goes to show, but let's talk about engagement where we have over there, interaction the CTA placement. We had engagement around best sellers visibility. We had engagements around search and the style and the behavior. We had engagement around offers with the shipping.
[00:27:16] Thresholds. We had engagement around orientation, the top nav and the filters around different styles because they sell apparel. Those are all experiments that were ran around engagement.
[00:27:27] SImbar: Yeah, those are good examples. How then do you prioritize those principles when designing and optimizing
[00:27:35] Khalid: So what you need to do again, you need to identify all the top different pages on the website. And I think, by the way, we should do a whole episode on prioritization, because we've gotten so much more sophisticated about prioritization. it really is a lot more fascinating to me right now. We used to have a very comprehensive framework for prioritization, where each of the problems that we've identified is scored against 18 different metrics.
[00:27:57] Nowadays, we score an item [00:28:00] against 8 different metrics, so less. But that's one step in our prioritization process. We start by identifying the financial impact of running an experiment on a page or a set of pages. If I ran an experiment, I hit this MDE, I ran for three weeks, what kind of impact? I expect a 14 percent lift.
[00:28:18] That translates to 200, 000. For this page, it's 150, 000. For this page, it's 50, 000. So right away, you can tell the financial impact for different pages on the website. I think that's very important because you want to focus on the pages that will have the most impact. That's number one. Number two, within the page, now you list all the different problems that you've identified using the conversion framework elements, the seven different elements.
[00:28:40] Oh, on the product pages for an e commerce website, I've identified 50 different problems, correct? And they show all the different elements of the conversion framework. Then we take each one of those 50 problems and we score it against The eight different elements off the prioritization framework that we have.
[00:28:58] this goes to how did we [00:29:00] identify the problem? Do we have data supporters? Do we have customer research that supports it? Is this easy to implement? So there's lots of things that we evaluate. Okay, so this is good. Now you have an initial goal. Prioritized list based on the page type and the impact of that page.
[00:29:15] So I know my PDPs will generate 200, 000. They are the top priority. Here's my first. Here are the experiments for the PDP. And then after that, maybe it's the homepage. And after that is the cart page. I'm just giving a random example. We then take each one of those problems and we start thinking about the solution.
[00:29:30] And we break the solution into, okay, do we expect this to be an iterative solution? Do we expect this to be a substantive or disruptive solution? And we want to have different coverage in each one of those areas. So I expect if I'm running a hundred experiments in a year, I expect to 60 percent of them to be what we call iterative.
[00:29:49] I'm just iterating on the current design. I'm trying to improve it. I expect about 30 to 35 percent of my experiments to be what I would call Substantial experiments, [00:30:00] correct? I'm introducing a completely new element to solve the problem and expect about 15 percent of my experiments to be what I call disruptive.
[00:30:07] I make sure that I have enough coverage around those, okay? That's interesting. And then I layered with another layer of saying, what is the complexity? of implementing the solution that I'm thinking about for this problem. And we classify the solution as low, medium, high. And we also want to have about 50 to 60 percent of our experiments within the low, about 25 percent to be in the medium, and about 10 to 15 percent in the high complexity experiments.
[00:30:35] Now that we actually have a ton of data around that, it's fascinating to us because you would think that the more time you spend on Implementing a solution, the higher the chance that solution has of generating a lift and revenue and conversion rates, that's not the case. I have enough data to show from 2000 even before that, and now we're adding 2024, that the amount of time that you spend Implementing a solution does not [00:31:00] relate to the impact that solution has.
[00:31:02] Sometimes something takes you about four to eight hours to implement that has such a huge impact on conversion rates. But I want to make sure that the experiments and the solutions that I'm coming up with have that coverage. So layers upon layers, and spent literally the morning looking at data from different clients trying to come up with different charts.
[00:31:21] Just think through because it's one thing when you think about one layer, but when you add four or five other layers on top of it, it's okay, you need to make the right decision. And there are just so many different factors, but it's extremely powerful.
[00:31:31] SImbar: That was good. Now I can't wait like for the prioritization episode, Which is coming out like next week am tempted to ask like some question about prioritization, but I'm also thinking that okay, cool.
[00:31:42] Let's focus on prioritization next week. So that like, uh, yeah.
[00:31:46] Khalid: Although I have to tell you next week, we're not going to have CRO live hour because I am going to be at PopCon Vegas. So it's the week that. We come back and we do the conversion framework. We will do the prioritization.
[00:31:57] SImbar: That's cool. So I guess those are the questions [00:32:00] that I had. For you today, unless there are any in them on LinkedIn.
[00:32:04] Khalid: I cannot see if there's any questions on LinkedIn. It doesn't look like there are any questions on LinkedIn. Thank you everybody who is listening to us, who's following us here. So we really appreciate your support.
[00:32:14] And until next week, happy
[00:32:16] testing.
[00:32:16] Outro: [00:33:00] Uh,