NBomber v5.8.0 - Scenario Weight, Runtime Thresholds

In the new release, we ship a few big features:

• Scenario Weight
• Native extensions for probability distributions
• Runtime Thresholds

Scenario Weight

Previously, NBomber allowed introducing randomness, but primarily within a single Scenario or Step. There was no native way to introduce randomness across multiple scenarios. In the new version we’ve unlocked a highly requested feature - Scenario Weight. The Weight value allows you to control the relative frequency or probability of Scenario's execution. Scenarios with higher weights will run more frequently during the test, while those with lower weights will execute less often, providing flexible control over load distribution. NBomber uses the weights to determine the probability of each Scenario.

Example: In this case, the "read_scenario" probability is 80%, and "write_scenario" is 20%.

var readScenario = Scenario
    .Create("read_scenario", ...)
    .WithWeight(80); // sets 80%

var writeScenario = Scenario
    .Create("write_scenario", ...)
    .WithWeight(20); // sets 20%

NBomberRunner
    .RegisterScenarios(readScenario, writeScenario)
    .Run();

For more information, please refer to our documentation.

Native extensions for probability distributions

In our documentation, we referenced several popular probability distributions: Uniform, Zipfian, and Multinomial. To use them, we noted that you need to install the MathNet.Numerics package.

To start working with probabilities, you should install MathNet.Numerics package. It's a free package that contains many useful probabilies.

In the new version of NBomber, we decided it would be beneficial to have these distributions available out of the box. Since they will be included with NBomber, we aimed to ensure they are highly performant compared to MathNet.Numerics. One downside of MathNet.Numerics is that its API caused extra allocations, which we wanted to avoid in our implementation.

• We added native support for Zipf distribution, which allows us to choose an item according to the Zipfian distribution. For example, when choosing a record, some records will be extremely popular (the head) while most records will be unpopular (the tail). The Zipf distribution is a mathematical concept that describes the frequency of items ranked from most common to least common in a dataset, like word frequencies in a book or city populations. Also, this distribution can be used to describe hot keys/items (skewed key distributions) in database or popular tweets in Tweeter web service.

var scenario = Scenario.Create("zipfian_distribution", async context =>
{    
    var stepNumber = context.Random.Zipf(n: 5, s: 1.3);

    switch (stepNumber)
    {
        case 1: // run step 1
            Step.Run("step_1", ...);
            break;

        case 2: // run step 2
            Step.Run("step_2", ...);
            break;
        
        ...
    }
});

• We added native support for Multinomial distribution, which allows us to specify probabilities for each item. For example, assigning a probability of 0.95 to the Read operation, 0.05 to the Update operation, and 0 to Delete and Insert. This setup creates a read-heavy workload. To simulate Multinomial distribution we can use extension method Choice for Random object.

// 70% for read, 20% for write, 10% for delete
var items = new[] { ("read", 70), ("write", 20), ("delete", 10) };

var scenario = Scenario.Create("multinomial_distribution", async context =>
{    
    var randomItem = context.Random.Choice(items);

    switch (randomItem)
    {
        case "read": // 70% for read
            Step.Run("read", ...);            
            break;

        case "write": // 20% for write
            Step.Run("write", ...);
            break;

        case "delete": // 10% for delete
            Step.Run("delete", ...);
            break;
    }
});

For more information, please refer to our documentation.

Runtime Thresholds

We're excited to announce that this feature has finally arrived for NBomber fans. Assertions and thresholds have been supported from day one. Typically, thresholds with assertions are applied after the active load test has finished, once all the available statistics have been provided. Here’s an example:

var scenario = Scenario.Create("http-scenario", async context => ...);

var result = NBomberRunner
    .RegisterScenarios(scenario)
    .Run();

// the test is finished, and we can now retrieve the final stats 
// to check against our thresholds.
var scnStats = result.ScenarioStats.Get("http-scenario");

Assert.True(scnStats.Fail.Request.Percent <= 5);  

However, there may be cases where you want to define thresholds that are active throughout the entire test run — we refer to these as Runtime Thresholds. Here is an example:

var scenario = Scenario.Create("http_scenario", async context =>
{
    ...
})
.WithThresholds(  
    Threshold.Create(scenarioStats => scenarioStats.Fail.Request.Percent < 10),

    Threshold.Create("step_1", stepStats => stepStats.Fail.Request.Percent < 10),
)

info

Runtime Thresholds, applied to real-time stats, help prevent the test from continuing execution if specific failure criteria are already met. Additionally, the results of these threshold checks are included in the HTML report, making them highly useful for performance analysis.

• Runtime Thresholds perform checks periodically, by default every 5 seconds.
• The results of thresholds checks are included in the HTML report.
• Runtime Thresholds can be defined in JSON Config.

The key feature of Runtime Thresholds is that they can be defined in the JSON Config. Generally, Runtime Thresholds offer flexibility in terms of configuration: you can set them to run without causing early termination of the test or delay the start of the threshold check.

"ThresholdSettings": [

    { "OkRequest": "RPS >= 80" },

    { "FailRequest": "Percent < 10" },

    { "OkLatency": "max < 100" },

    { "StatusCode": ["500", "Percent < 5"] },
    { "StatusCode": ["400", "Percent < 10"], "AbortWhenErrorCount": 5 }
    { "StatusCode": ["200", "Percent >= 90"], "StartCheckAfter": "00:00:20" }
]

After completing a test that includes Runtime Thresholds, the results of the threshold checks will be available in the HTML report. This allows you to easily review and analyze how the system performed against the defined thresholds throughout the test run.

For more information, please refer to our documentation.

Get/Find/Exist extensions for Stats data

The Stats Extensions include the useful helper methods: Get, Find, Exists, which are used to work with ScenarioStats, StepStats, StatusCodes, and other similar data.

var result = NBomberRunner
                .RegisterScenarios(scenario1, scenario2)
                .Run();

// gets scenario stats: 
// it throws exception if "scenario_1" is not found 
var scnStats1 = result.ScenarioStats.Get("scenario_1");

// finds scenario stats: 
// it returns null if "scenario_2" is not found 
var scnStats2 = result.ScenarioStats.Find("scenario_2");

// check that step "login" exist in the list
bool isLoginExist = scnStats1.StepStats.Exists("login");

// gets "login" step stats
// it throws exception if "login" is not found 
var loginStats = scnStats1.StepStats.Get("login");

// check that status code "503" exist in the list
bool isExist = scnStats1.Fail.StatusCodes.Exists("503");

// get stats of status code "503"
// in case of status not found, the exception will be thrown
var statusCode = scnStats1.Fail.StatusCodes.Get("503");

HTML report improvements

We’ve improved the HTML report by updating the logic behind the latency indicators panel. Previously, this panel displayed cumulative results across all scenarios. Based on user feedback, we adjusted it to show data specific to the selected items. For instance, if you select a specific step to analyze, the panel will re-render to display data filtered specifically for that step or scenario.

Example: Scenario latency indicators panel.

Example: Step latency indicators panel.

Disable rendering of Metrics Table in Console

In certain cases, some of you may have noticed that using the console's stdout for debugging can be tricky, as the NBomber Metrics Table may overwrite your printouts. If you want to use the console for debugging purposes, you can now disable the rendering of the Metrics Table in the console.

In code:

NBomberRunner
    .RegisterScenarios(scenario)
    .DisplayConsoleMetrics(false)

Also, via CLI args:

dotnet MyLoadTest.dll --display-console-metrics=false

Also, via JSON Config:

{
  "GlobalSettings": {        
    "DisplayConsoleMetrics": false
  }
}

LoadConfig improvements

We have a method that loads JSON configuration. Previously, this method only supported loading data from a file or URL. We've extended it to allow loading JSON content as a string.

NBomberRunner
    .RegisterScenarios(scenario) 
    .LoadConfig("{ YOUR JSON }")

What is next

• Release Notes for v5.8.
• Please follow our Roadmap page for the upcoming release.