Skip to main content
Our docs redesign is live!
Guide: Lytics use cases and technical implementation guide
•
4 min read
seatter-andrew.png
AuthorAndrew Seatter
huiza-eric.png
AuthorEric Huiza
TKLUV7DCL-U07KCKW9HE2-7f4573f106b6-512.jpg
AuthorLo Etheridge

Use case 1: A/B testing

Why does this matter?

A/B Testing and Multi-Armed Bandit (MAB) distribution, orchestrated by Contentstack Personalize and fueled by Lytics data, ensure that every user interaction is an optimized opportunity. Use A/B testing for decisive, statistically robust outcomes on major changes, and leverage MAB's continuous machine learning to dynamically maximize real-time conversions and revenue from high-traffic elements, guaranteeing that the best-performing experience is always prioritized.

What is an A/B test? What is MAB?

Concept

Description

When to Use

A/B Test

Compares many versions of a single variable, typically an element on a webpage, to determine which performs better against a defined goal.

When you have a strong hypothesis about a specific change, want definitive statistical proof, and the experiment needs a set duration or sufficient sample size.

Multi-Armed Bandit (MAB)

Machine learning approach that continuously allocates traffic to the best-performing variation during the experiment, favoring the variation that is showing the most success.

When the cost of showing a sub optimal experience is high, you have a shorter window for testing, or you want to maximize immediate conversions/rewards while learning.

When to use A/B testing or MAB?

  • Use A/B Testing when you have a strong hypothesis, need definitive statistical proof, and have a set experiment duration. Ultimately, use A/B testing when you need to make foundational changes, high-stakes decisions, or when you need a clear, statistically sound winner to deploy permanently.
  • Use MAB to continuously allocate traffic to the best-performing variation. Ideal for high-traffic elements (like hero banners) where you want to maximize immediate conversions while learning. When you want to optimize high-traffic, continuous elements (e.g., product recommendation slots, hero banners) where maximizing real-time performance is critical, use MAB.

How to activate

  1. Navigate to your Contentstack Personalize Project: Click the New Experience button in the top right corner.
  2. Select Experience Type: Select A/B test from options of experiences.
  3. Name A/B test: Name your A/B test and save the general details on the bottom right-hand side of the screen. View creating A/B test documentation.
  4. Select Variant Distribution: Navigate to Configuration on the left hand side of the screen. Select traffic allocation (e.g., equally split, MAB, custom settings).
  5. Create Variants: Below the Variant Distribution option, click the + Add Variant button. Name your Variant and add the rest of your test cases the same way.
  6. Add Event to Track Conversion: Once you're finished adding your variants, add an event that is tied to the A/B test to track conversions (if using events, one will need to be created before creating a segmented experience). View Events Documentation.
  7. Choose a Target Group: Below the Events section, select a target group. There are 2 options: Everyone or Selective. If you choose Selective you can target a specific audience that you've built in Lytics. Once that is done you can save your draft.
  8. Link Content Types: Now that your draft is saved we can navigate back to the CMS and into our Stack. Once inside your stack navigate to Settings, scroll down on the left hand side of your screen till you see Variants as an option and click the variants button. Youwill see a menu of all of your created experiences, select your newly created A/B test. Once inside the CMS settings for your A/B test select the content types you wish to link to your variants. These content types will act as the personalized content for your A/B test.
  9. Create Personalized Content: Create an entry for the content type you associated with your A/B test in the previous step (e.g., Homepage, Landing Page, etc). Publish base content for this entry, once that's done click the button near the top right hand side of the screen that says [Base Entry]. You will see a dropdown appear with the variants you created. Select the variant you wish to modify, once you've edited content save and publish your entry to your desired environment. (Using Live Preview or Visual Builder is best practice to visualize your changes when editing variants.) View Creating Personalized Content Documentation.
  10. Go Back to the Experience and Activate it: Now that you've created an A/B test, created variants, linked them to content types and created entries for your variants it's time to activate your A/B test. Navigate back to your Personalize project and click Configuration on the left hand side of your screen. At the bottom left hand side you will see the Activate Draft button, click that button and your A/B test is now live! You can track impressions and conversions via the analytics sections of your A/B test below the configuration tab. View Experience Analytics Documentation.
  11. Depending on your front-end framework, you will need to use a different SDK to retrieve and initialize your personalization experiences. Here is an example using Next.js.
Validate (recommended): After activation, confirm (1) impressions are incrementing, (2) conversion event fires, and (3) analytics shows variant performance.