Elite Experiments at Experimentation Elite with Jonas Alves

No hacks invited to the party. No hacks invited to the show. We've got a good thing going. No hacks invited today.

[00:00:18] Sani: Welcome to no hacks show, a weekly podcast in which smart people talk to you about better online experiences. This is another episode. No hack show is doing with experimentation. Elite experimentation lead is the UK's premier experimentation and zero experience taking place on June 11th and 12th in Birmingham, UK.

So head over to experimentation, elite. com and get your ticket. Now, today I'm joined by someone who has built a platform that helps you create a better online experiences. Jonas Alves, CEO and co founder of AB smartly. Jonas, welcome to the podcast.

[00:00:48] Jonas: Thank you, Sonny. Happy to be here with you.

[00:00:50] Sani: Pleasure to have you on. Pleasure to have you on. So, uh, we're going to talk about what you did with AB Smartly, uh, about why you chose to partner with Experimentation Elite and why people should go to the event in the first place. But before we start talking about that, I have a question for you. So I'm going to put you in a scenario. You want to hire a new experimenter for, for AB Smartly, let's say. The problem is the economy is. Flawless and awesome. And everybody has a job and there's no humans available. You have to hire one animal for your team. Which animal is it and why?

[00:01:24] Jonas: Well, which animal will we will hire for the team, right? We'd like to be super supportive, like customer services on top of our minds. So it has to be an animal that is super helpful. Maybe a horse, uh, someone that, uh, actually at booking. com, we would say that, uh, I worked many years for booking and booking would say that we are workhorses and not show ponies.

You know,

[00:01:52] Sani: you go. There you go.

[00:01:55] Jonas: yeah, uh, I would, I would

[00:01:58] Sani: A workhorse.

[00:01:59] Jonas: work a lot.

[00:02:00] Sani: I love that. That is so good. So, uh, now that we hired a horse, uh, to run some experiments, tell me about, uh, your AB testing and experimentation journey. How did you get

[00:02:11] Jonas: Yeah, basically I started with A B testing when I joined Booking. com in 2008. Booking was one of those first companies, uh, like early, uh, into experimentation. Uh, they were running experiments since 2005. Um, It was quite basic when I joined the team. And you know that booking is like they have the fame of being huge experimenters.

But when I joined, it was just the website team running experiments, and it was a small team. Actually, when I, when I joined the team, uh, Jasper, that was my boss, well, he was a developer, but he was moving to a full time management position. So I became the developer of the team, and it was basically myself and Luciano, the designer running experiments in the website in the beginning.

So very small team, but the tool was enough to do what we could see the conversion rate per variant and we had a chi squared test for significance and that was enough to run a few tests, right? It was actually during those times that we run some of the biggest successes for Book. com in terms of A B testing.

But we started, uh, uh, uh, uh, needing better tooling evolving, uh, uh, becoming more sophisticated, right? So I built a proper experimentation platform for booking. com. I made it very easy to track new metrics with one line of code. Basically, we built a CDP. We could add a track cancellation, track newsletter subscription, track clicking a button.

At some point, we had like 5, 000 metrics being tracked across the board. I became the head of experimentation at Booking, so responsible not only for the tooling part, but also for Training for new hires like booking was a hiring like crazy. Like we were doubling the number of developers pretty much every year and I felt that that culture of experimentation was becoming quite diluted across the organization, right?

We had more new people than than the old ones there and the so I I made a huge push for for training So I I built training for new hires training for already seasoned experimenters that were already running experiments for a couple of years um Made it better with the statistics as well. I reached out to Ronnie Kohavi at Microsoft in 2011 or 2012.

Ronnie, you probably know about Ronnie, right? He wrote that amazing book called The First Worldly Online Controlled Experiments. That's like the Bible of A B testing experimentation. Ronnie referred me to a colleague of him that worked with him both in the platform at Amazon and later at Microsoft and that person helped us a lot with the statistics of the tool as well.

Ronnie. And then in 2015, when, uh, when I decided to leave the company and pass the reins to Lucas Vermeer, that continued with the, with my role at booking. com, uh, booking was like a factory, right? We were putting in production between 100 to 200 experiments per day. So every single day, 100 to 200 experiments we're putting, uh, we're being put live.

Uh, and all they said more than 1000 running across the board. So really crazy. Um, So I learned a little bit about experimentation and I started consulting as well. So I, I joined trip in a year as a, as a co founder. I worked first one year with, with Katawiki to help them build their own experimentation platform.

And we built one for a trip in here as well. Uh, uh, together with the, uh, my co founder now at AB smartly, Marcio that also worked at book. com. So basically you move there first and then I

[00:05:36] Sani: all started there at booking. com. All roads in experimentation lead to booking. com. If I couldn't count the number of times that a guest on this podcast has said, you know, if you don't have a program like booking. com, like it's still the example that people use for a successful high velocity. Huge, basically experimentation program.

And that is, I think the greatest testament for what you and the rest of your colleagues did, did in the early days and not just the early days at booking. com is just, yeah. And even for me, I have not booked a hotel or accommodation anywhere else for seven or eight years because of how, how good that system is, how well it knows me, how well I know it.

So what you did there is absolutely, absolutely magnificent. And that, that, that led you to experience building experimentation program, programs led you to want to build your own, right? Right.

[00:06:31] Jonas: AB smartly. Right? So I, I. With my, uh, consulting, uh, customers, clients, pretty much all of them were either, uh, uh, building in house or, uh, they, they were using a third party tool like, uh, Optimizely, those, uh, uh, those, uh, uh, like the first A B testing tools that, uh, that actually, uh, came up to the market.

Uh, But when they reach a specific sophistication level, that's not enough for them anymore. Right. They need the tool. They were basically help asking for my help to build on top of the tool or to help them build in house from scratch. Right. So I noticed that I'm helping every company doing pretty much the same stuff.

Right. So there's there's something that is missing in the market here. And that's why we built the company. Basically, they were always looking for a tool with a knowledge base with that would help them with the Different stats engines, like different statistics, sometimes for, uh, uh, uh, SEO experiments or a diffs on diffs, whatever, right?

Uh, easy collaboration and sharing of results with other teams, uh, uh, across the company, right? And, and the tools that were there in the market were more focused on, on, on marketing teams, right? On how to go around the development and, uh, and easily make changes in the website.

[00:07:47] Sani: Let's just say Google optimize. Let's, let's say Google optimize. No, but, but I think they did a lot of damage as well. Tools like that, where you can click and drag and drop and create your experiments without need to know, or use any code and. They were not always great. Let's just call it that. And, and, and, and Google this optimized discussion, just like Google did. Uh, so one of the taglines, or it's a meta description tag of AB smartly in the homepage is run AB tests, 20 percent to 80 percent faster. How can you explain that?

[00:08:26] Jonas: Yeah. So basically that's because of our group sequential testing engine, right? Group sequential testing

[00:08:32] Sani: Mm.

[00:08:33] Jonas: of both worlds in between fixed horizon and fully sequential testing, right? Most of the tools in the market use fully sequential testing. Uh, you probably know the story from optimizely, right?

When they came to the market, they were using fixed horizon, but they were not teaching the users on how to do power calculations. So they didn't tell them that they actually. Needs to make a power calculation, make a decision at that point in time. So they were making decisions at any, as soon as they would see significance, basically, and that, so the false positives were through the roof.

So they were quite precise because of that. And then they came up with the, the, the fully sequential testing engine, uh, that basically, uh, saves guards, uh, it's, basically they say that, uh, you can pick at any time, uh, without raising the false positive rate. And that's true, uh, uh, uh, uh, uh, in the majority of the cases, right?

Uh, the thing is that you lose power with it as well. So for, for experiments that don't have a huge impact, like, uh, uh, with less than 5 percent of the, of impact, which is the majority of the cases, if you are already, uh, amid your, uh, uh, website, uh, then it takes much longer than FixHorizon, right? Uh, So with group sequential testing, you get the best of both worlds because you, you get power very close to fixed horizon, but still with the flexibility of making decisions before reaching the full sample size.

Right, so you can configure it with a, maybe I want to have one, uh, uh, checkpoint per day, so I can make it, at least I can make a decision once a day, right? Or maybe, uh, I'm sure that I will need to run the experiment once. For at least one week. So during the first week, I don't have analysis. And after the first week, I start having analysis every day, right?

So you can configure it as you think that it works best for you. Uh, but because every time that you make, build a checkpoint, you pay in power, then it's best to use an approach like this versus fully special testing, where basically you, you, you pay for every new visitor that is added to your experiment, right?

So, Basically with every exposure, uh, you are paying in power for that, right?

[00:10:46] Sani: So pretty much a best of both worlds is, is what you were, what you set out to build, had set out to build with AB smartly. And I mean, their testing platforms, again, the one Google had, it was, I don't know, it was a gateway drug to testing for most people. And that's why it was good for, for some reasons, but also it, it, it was, Taught people how to do it incorrectly, in my opinion, because it's, it's never that easy.

So there's that there's a platform that you slap on your website, that it, there it is, do the testing. And then there's the building of your whole experimentation program, like what you did for booking. So AB smartly is like somewhere between those two worlds, you get the customization, but you also don't have to build everything because I mean, you and your team have already done it.

Obviously. So, uh, we need to talk about a little bit about the future of A B testing, the future of experimentation. And we're recording this in April, 2024. What do you think the biggest developments in the world of experimentation will be in the next, let's say, 12 months?

[00:11:49] Jonas: Yeah, I think that that's that's a question. Focus on on a I probably right. It's

[00:11:55] Sani: every question in April, 2024 is what should I have for breakfast? Take a photo of your fridge and ask that GPT that every question is AI. Yeah,

[00:12:05] Jonas: for us and it will help with pretty much everything, right? So I think that I will help a lot with it, right? Uh, you know, Craig Sullivan. Right. He actually just shared these, uh, ebook about, uh, uh, AI prompts for, for, uh, um, maybe testing right for experimentation. Uh, and I think that in the future, all the tools will have that, right?

So, so basically

[00:12:29] Sani: you mean built in, built into the

[00:12:30] Jonas: built in, right? So the tools will actually analyze your previous experiments, come up with ideas for new experiments, help you make better hypotheses for, uh, for the experiments. Right. Help identify insights into experiments that maybe a person would not notice and the I can right.

So it will basically be an experiment on steroids, right? So it will give you all that extra, uh, help and knowledge that it's difficult, especially for smaller teams in the beginning that don't have a lot of experience like there will be. I think it will be a huge gain in terms of velocity and and and Insights and learnings.

So people will learn a lot faster with AI helping you with, uh, uh, with all of that, right?

[00:13:17] Sani: So basically like a, almost like a team member that anybody can talk to, ask questions

[00:13:21] Jonas: A very experienced team member pushing the team forward.

[00:13:24] Sani: Yeah. Pushing the team forward or taking them off a cliff. Like we, we still don't know what, what the end result will be, but I like that. Uh, a few other people have told me that on this podcast as well, basically they see AI as it is today, as.

A super smart and super fast team member that you don't really have to pay more than 20 a month, which is wonderful. And you can ask all the questions it can. It's much faster than anyone else can be on your team. I think that's a good, that's a good situation. We're at a good stage with the AI where it is in, in like five years, God knows, but, but we'll see also that the Craig Sullivan book, I have to Give him a call.

You want to run down there and Marcella, uh, to shout out an incredible book. I was lucky enough to get an advanced copy. Absolutely amazing. I even left him a testimonial. It's on, on the website. I couldn't believe they put me right there with Ruben DeBoer and Michael Ogar. So. Very honored to be, to be there.

[00:14:19] Jonas: I saw your testimonial there as well.

[00:14:21] Sani: It's amazing, right? Uh, it, it's, it's very good. Like I said, in, in that testimonial, you can get results in, in a few minutes. You just find the playbook you're, you're looking for, let's say, create slides for a presentation. It's a few minutes from, from starting reading of the playbook and, and getting results with AI.

That, that's how efficient it is. So check out that book. It's free. What does, I mean, Go get it. Uh, okay. Uh, let's talk about like the big event that's happening in June, the, the premier event happening in the UK, where we will both be this year, experimentation elite, first time, uh, that it's not in London, it's in Birmingham.

It's first time that it's going to be two days, what I hear. And from, from what everyone's saying, this is going to be the event of the year for experimentation in the UK, at least. So tell me about AB smartly and tell me about experimentation elite. What's the link there.

[00:15:14] Jonas: Yeah. Basically, I think that the name, uh, has something to do with it as well, right? We believe that we are a tool for elite experimenters. So it makes sense to partnership with the exploitation elite. But, uh, but it's also like, uh, uh, we have been there for, uh, uh, uh, the last two, uh, episodes or, uh, say the last two events.

Uh, and it's great to see old friends and to make new friends. Like the people are amazing, right? So, uh, uh, I really like to be there just to talk with people and, uh, share information. Share, uh, uh. Uh, like having nice conversations. Um, and it's also nice like to, to share about, uh, uh, uh, find of course, uh, uh, companies that might be willing to, uh, know more about AB Smart and learn more, uh, about our tool and eventually became customers, right?

But, uh, it's more than that. Like it's, uh, uh, I think that's, uh, uh. All the community is, uh, is amazing. And I really like to be there just to talk with people and, uh, and making friends, like, uh, I think that's the most important part of

[00:16:24] Sani: I think that is one of the greatest aspects of the experimentation world and bubble. Let's call it bubble. The community is just. Incredibly welcoming. There are no bad apples. Everyone's nice. Everyone's open to helping. If you just raise your hand and be friendly in the way you ask your question, it's a very unique community from from from everything I've seen.

Now, of course, communities happen online as well. Especially since the pandemic people talk online. I mean, we all have linked in that we're doing this online. But what is the differentiator between online, uh, networking and communities and in person events? Like this is like the big thing with things like experimentation, but all the other experimentation conferences out there.

[00:17:11] Jonas: Online is great, right? Because it's, you don't need to go outside and you can do it every time. Like I left the TLC channel, right? And, uh, I love to be there and answering questions and, uh, uh, just reading, uh, uh, about, uh, what is happening basically in the, in the community, uh, Face to face is different, right?

It's always nice to be there with friends, having a nice conversation, talking about experimentation or anything else, right? So, uh, I don't think that online replaces that. I think that online is great and it helps. And it's, uh, but you need both. You need to socialize with people as well, right? Especially now where everyone works from home, right?

It's always nice to sometimes you need to socialize. Go outside and meet real people instead of being always in front of the screen, right? So I think that's the biggest

[00:18:06] Sani: The fact that you said you need to socialize with people and it doesn't sound weird at all in 2024 tells a lot about our society and how maybe we did it wrong at some point, but that's just where we is. Most of our lives are online. I mean, it, it, it just how it works. Now, uh, one more question about the future of experimentation, AB smartly or any other tool.

Do you think there will ever be a time where a website can just run its own experiments using AI? So there's no human. Intervention at all, we think we'll, and then implement those fixes. Like, do you think that will happen? Let's say in the next five, 10, whatever years.

[00:18:42] Jonas: Well, if you believe that A. I. S. Will be able to build full apps from scratch, then why not just optimize them as well? Right? Uh, I think it's possible, right? And maybe we're not too far from it. So let's let's wait and see.

[00:18:57] Sani: think the, the, the craziest thing about AI is whatever use case anyone mentions, the answer is, That could be closer than we think. Like that just tell me, because before opening, I announced Zora, like we said, Oh, in a few years, there's going to be videos in it tomorrow. Like, look at this. We built a video making tool and now you can make a movie. Like, okay. Uh, okay. So, so for people who want to try AB smartly today, who want to sign up and start using it, what is the best way? absmartly. com obviously is a website. The link will be in the description, but what is the best way to get started with AB

[00:19:31] Jonas: Either reach out to the website or just reach out to me on LinkedIn. Um, yeah, it either works. We can, we can set you up on a POC so you can test the website and, uh, yeah, just

[00:19:44] Sani: It's a very easy onboarding, uh, for, for,

[00:19:47] Jonas: Usually we on board companies, uh, we get them running experiments in a one to one, one day to one week. Right.

So it really depends on, on the All how much, uh, they would like to, uh, uh, instrument, right? If they have just a single website, usually like in one hour they are up and running. But if they want to instrument also the iOS app, the Android app, uh, the microservices, right? So, uh, email, uh, uh, the email newsletter, then it might take a bit longer.

[00:20:12] Sani: And the integration, the, the, the installation of the platform, you help them with that, obviously. Right. Okay.

[00:20:19] Jonas: we basically, uh, do that together. Uh,

[00:20:21] Sani: You do it together with the, with the user, which is perfect. Uh, it is completely platform independent from what I understood there.

[00:20:27] Jonas: Yeah, so that's the idea, is that we work with any stack, right? And, uh, um. And, uh, whatever, whatever the data is, uh, we, we got it right in the, we ingest that data and we use it for, uh, metrics and segmentation, anything, you know,

[00:20:43] Sani: Sounds simple enough. Uh, so, uh, absmartly. com or your LinkedIn, which will be in the description, is the best way to get in touch. Also, Experimentation Elite, we need to end with a call to action. Uh, uh, You should go to ExperimentationElite and you should go to ExperimentationElite. com, get your ticket, meet Jonas, meet all the other wonderful people, I think there's, I'm gonna say 20 previous guests of this podcast who will be there, who I cannot wait to meet in person, so ExperimentationElite.

com, get your ticket, Birmingham, June 11th, June 12th, see you there. And to everyone listening to this podcast, I want to thank you, especially you Josh, for being a guest. I want to thank you for, uh, for listening. Please consider rating, reviewing, and sharing the episode, and I will talk to you next week.