Running promotions - in the future

It has been immensely useful during this year’s black week. Here’s how we approached it.

If you just want to look at the code, go ahead and scroll down!

I work with a client that has thousands of promotions. Most of them are not active, but there is still an impressive amount of active promotions. Here’s some of the reasons as to why I recommend against ever deleting a promotion that has been in use. It’s very fun and challenging to find all side effects with this and we’ve had our share of bug fixes and/or feature requests from this client that’s been embedded in Epi Commerce.

The problem

Running a B2C ecommerce site can be a lot about promotions and in our case it is. They run different promotions now and then, which might affect big chunks of the assortment, sometimes the entire assortment. Of course they want the prices to be adjusted as soon as the promotion becomes active, which the promotion engine and the cart solves wonderfully. But when it comes to the price in product listings, it is not viable in real-time to run the promotion engine for every product in the product list, hence we use an indexed value. This means we have to run a product indexer before the prices are changed. As a full index currently takes 2h for 200k+ products, the prices on product listing pages are shown 2 hours after the promotion is active.

When the business model relies on having scheduled promotions in marketing (an example: 20% off everything starting Black Friday 00:00), it does not thrive when customers see the wrong price in the listings for several hours.

So me and my colleague Ola Sandström put our brains together and theorized a solution. What if we can prepare a result of an index with prices coming from a future timestamp? So when that time comes, we can simply push all of the pre calculated prices to the product index, having the heavy lifting already done in the past.

There is one big risk with this approach, that the prices we pre-calculate mismatch what the real prices would be, if we spent the 2 hours at the time and figuring out what they would be. We tried to outline every scenario this would happen:

• The price of a SKU is changed in any way, after we’ve done the pre-calculation but before the time set for the pre-calculation.
• Any promotion has their data changed that would affect the price in the same time span as above. Note that you can set up the entire schedule and promotion structure ahead of time with ValidFrom and ValidUntil.

Both price changes and promotion changes can be carefully managed, so we and the client agreed that these risks could be entirely mitigated with simple human instructions. But to be safe, we never entirely trust our pre-calculated value and we start a normal indexing just as we did before and when that is done it will flush those changes to the product index. So there is a window of around 2 hours where we show the pre-calculated prices and after that the normal indexing job catches up and overwrites, hopefully with the exact same data.

To make this all work, it requires the following components

• A way of running the promotion engine pretending to be in the future
• A way of storing and managing pre-calculated price lists connected to a certain timestamp
• A way of only changing some properties in an indexed product while keeping all other values the same. This is needed so other changes to the indexed product can happen and be reflected, even after we’ve applied the pre-calculated price changes. Stock, content or any other change not related to discounts or prices should not be stale because of the pre-calculated price.

In this blog post, I’ll only talk about “A way of running the promotion engine pretending to be in the future”. I might add more posts to cover the other parts as well.

The solution

Essentially, we need to do something like this (pseudo code)

using (pretendToBeInTheFuture) {
   var rewards = _promotionEngine.Evaluate(a product)
}

Which is exactly what we did! (real code :D)

using (new RunPromotionsAtSpecificDate((DateTime) timestamp))
{
    var rewards = _promotionEngine.Evaluate(skuReference);
    CalculateAndStorePrice(skuReference, rewards, timestamp);
}

I found out while digging in the promotion engine that one of the first steps is to load all promotions that could give us a reward (if you are interested in knowing this process I wrote about it in a forum thread). This is where we filter out promotions and campaigns who has a start date that is in the future, hence not active for us. The class responsible is PromotionEngineContentLoader and method is GetEvaluablePromotionsInPriorityOrder(IMarket market). This method (aside from adding caching layer) calls a private GetEvaluablePromotions(IMarket market). This method essentially does this:

var campaigns = _contentLoader.GetChildren<SalesCampaign>(GetCampaignFolderRoot())
    .Where(c => _campaignInfoExtractor.IsCampaignActive(c) && IsValidMarket(c, market))
    .ToDictionary(x => x.ContentLink);
var promotions = GetPromotions(campaigns.Keys)
    .Where(x => IsActive(x, campaigns))
    .OrderBy(x => x.Priority)
    .ToList();
return promotions;

What we see is that somewhere in here, we call a sub function that will filter based on the current time, but for both promotions and campaigns. So we need to make sure we do the same logic for both. IsActive will in the end call GetEffectiveStatus(promotion, campaign) on the same CampaignInfoExtractor. And both of these calls will ultimately call GetStatusFromDates(DateTime validFrom, DateTime validUntil)

As you might figure, this is where we want to change the logic. So what we want to do is to inherit this class, make Structuremap use this class whenever it’s injecting CampaignInfoExtractor and only override the GetStatusFromDates method by so:

private override CampaignItemStatus GetStatusFromDates(DateTime validFrom, DateTime validUntil)
{
    var now = GetNowOrFutureTimestamp();
    if (validFrom > now)
        return CampaignItemStatus.Pending;
    return !(validUntil > now) ? CampaignItemStatus.Expired : CampaignItemStatus.Active;
}

And the added thing and magic here is the newly added GetNowOrFutureTimestamp() call. To figure out what that method should look like, we need to look at where we store the future timestamp, which was the disposable RunPromotionsAtSpecificDate mentioned earlier.

This is how we built it

public class RunPromotionsAtSpecificDate : IDisposable
{
    public RunPromotionsAtSpecificDate(DateTime dateTime)
    {
        PromotionEngineContentLoaderDate.DateToExecuteUtc = dateTime;
    }
    public void Dispose()
    {
        PromotionEngineContentLoaderDate.DateToExecuteUtc = DateTime.MinValue;
    }
}

With a supporting class:

public class PromotionEngineContentLoaderDate
{
    public static DateTime DateToExecuteUtc { get; set; }
    public static bool HasSetDate => DateToExecuteUtc != DateTime.MinValue;
}

As you see, we use DateTime.MinValue to represent that there is no future value set, that the GetStatusFromDates should use the “real now ” value, as it worked before. We also use static values to have a global accessible value. This might not be optional. We only took this simple route because:

• All our jobs run in an isolated service.
• Only 1 job is responsible for running the promotion engine, so this static value would only be used for that job
• We run multi-threaded, but all threads are either working with the same future time or the present time, never parallel threads working with different promotion work sets.

If you use this solution, make sure you don’t get any multi-threading issues and make sure the logic of future is isolated to the task you want to run in the future, nothing else. 

Now we can create the simple GetNowOrFutureTimestamp() method that would look like

return PromotionEngineContentLoaderDate.HasSetDate ? PromotionEngineContentLoaderDate.DateToExecuteUtc : DateTime.Now;

And that’s it. Very simple in the end, but has been an extremely powerful tool for us and our client with all the different major assortment price changes during black week and cyber monday. The code is found below. If you have any questions don’t hesitate to ask! And let me know if you are interested in the other topics as well that made the entire solution:

• A way of storing and managing pre-calculated price lists connected to a certain timestamp
• A way of only changing some properties in an indexed product while keeping all other values the same. This is needed so other changes to the indexed product can happen and be reflected, even after we’ve applied the pre-calculated price changes. Stock, content or any other change not related to discounts or prices should not be stale because of the pre-calculated price.

The code

Not much code, but very powerful depending on what you want to do with it!