Developer Guide

  • 2021.2
  • 06/11/2021
  • Public
Contents

Create a Workload Validation Script

The data streams optimizer tool helps you automate platform tuning for individual data streams by mimicking typical real-time actions, such as:
  1. Tuning the platform.
  2. Running real-time and non-real-time workloads on the system, and inspecting platform performance to confirm successful tuning.
  3. Applying different tuning settings and repeating the tuning process to correct tuning.
The workload validation script is a program that encapsulates step 2 and that is specific to your workload and your performance requirements. The data streams optimizer runs the workload validation script to verify whether the applied configuration meets requirements in the pre-production environment and thus confirm if the tuning for the platform was successful.
You are responsible for creating the workloads and writing the workload validation script.

Specifications

Your workload validation script must meet the following specifications:
  • Runs your real-time workload.
  • Runs your choice of interference (noisy neighbor) to match production behavior.
  • Checks whether your workload’s requirements are met.
  • Returns an exit code of 0 if the requirements are met and 1 if the requirements are not met. The tuning process stops if the returned value is not 0 or 1, indicating a validation completion failure.
  • Resides on the target system.
Your script can validate whatever performance requirements you have defined for your workload. These requirements can differ from data stream requirements defined in the requirements file.
For example, your script might validate the total cycle time instead of data stream latency. The idea is that your performance requirements are what is important to you, while the data stream requirements are what is important to the functioning of the tool. The latter benefits the former.
The tool accepts any type of program as a workload validation script. Example of Data Stream Latency Validation shows a Python* program.
To allow the tool to run the workload validation script, you must enter the script command and arguments in the requirements file.

Generating a Workload Validation Script

A variety of techniques are available to confirm the platform performance including, but not limited to, any combination of:
  • Running the real-time and non-real-time workloads on the platform and inspecting the workload logs.
  • Inspecting performance counters, such as CPU utilization, memory utilization, and I/O bandwidth, to confirm that the platform isn’t oversubscribed or undersubscribed, or skewed in load distribution.
  • For user-interface intensive workloads, such as those requiring manual input and plotting graphs, you might interactively confirm a lack of delays or glitches in the input or output operations.
Workloads can have cycle times that vary from microseconds to milliseconds, and are typically executed millions or even billions of times in a preproduction environment. These executions confirm that the platform performance meets expectations and any exceptions are within acceptable tolerances. Since real-time workloads are expected to execute continuously without an end point, developers typically use scripts to automate performance validation. The scripts run the workloads and mine the collected logs and/or performance counters. Without automation, the tasks to inspect the output of millions of executions of workloads and confirm platform performance would be manual.
The workload validation script is expected to execute all real-time and non-real-time workloads on the platform in parallel and for as many iterations as needed, possibly millions or billions, to confirm that the platform performance meets expectations, and that any exceptions are within acceptable tolerances. Automation to run test workloads may already exist to analyze your system, and you may only need to create a script to read those results. The script would inspect any needed logs, performance counters, and other metrics that are already collected, and provide confirmation to the data streams optimizer if the tuning was successful.
You might manually observe performance confirmations based on interactive experiences, such as for graphics performance. From your observations, decide if the tuned platform performance meets your expectations. For interactive evaluations, your script could, for instance:
  • Inspect a text file that you create upon completing the interactive inspection.
  • Use another method to provide manual input of the pass or fail result based on your criteria.
  • Try a different tuning setup, even if the results meet your expectations, and repeat the validation process.
If the tuning was unsuccessful, the tool would iteratively attempt to more aggressively tune the platform until a successful tuning can be identified and confirmed, if such a tuning is available.
Use these workload validation script design tips when reviewing your current automation, or when testing the platform performance with a new script:
  • The script should execute all real-time and non-real-time workloads to represent a production scenario. The profile of these workloads should represent expected modulations in production, such as increased/decreased consumption of resources and interference patterns.
  • The script should collect all needed measures to determine success of tuning. These measures may include performance counters and logs from applications.
  • Where interactive input is needed to determine the success of tuning, such as GUI application performance, the script may introduce user-interface prompts and pauses to collect user inputs.
  • The script should parse through all collected measurements and judge the tuning success to communicate success or failure to the data streams optimizer.
  • For a scenario in which complete system performance profiling with various tuning settings applied is desired, the workload validation script should collect measurements for every tuning applied and return failures to the data streams optimizer no matter the actual outcome. The measurements can subsequently be analyzed to determine which tuning attempt resulted in optimal performance. A correlation with the data streams optimizer logs for that tuning attempt yields the exact requirements to be provided in the input requirements file for obtaining that desired optimal performance, which you can subsequently run and verify.
  • You may choose to not create, or may be unable to create, a workload validation script, and instead manually observe the target platform performance to confirm tuning success. In such a scenario, the workload validation script may simply consist of a “pause” command, which would allow you to perform needed observations and then return a success or failure signal to the data streams optimizer through the prescribed interface.

Example of Data Stream Latency Validation

One type of validation is to measure the maximum latency of the data stream being tuned and compare it to the latency value in the requirements file. If the measured latency exceeds the latency requirement, the validation fails.
The provided MMIO Read Latency (MRL) test workload and associated workload validation script demonstrate a method of validating the latency of the core-from-PCIe (MMIO read) data stream. The test workload is a C application. The validation script is written in Python.
The following diagram shows the interaction of the test workload, workload validation script, and data streams optimizer tool.
  1. The MRL workload validation script runs the MRL workload with options specified at the command line. Code snippet from the validation script:
    # Run workload print("Start validation") command = f"taskset -c {args.core} {VALIDATION_WORKLOAD} "\ f"-a {args.address} -i {args.iterations} -p {args.processor} -o {args.outliers}" result = subprocess.Popen( shlex.split(command), stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True ) result.wait()
  2. The MRL workload validation script waits for the workload to run and generate latency data in nanoseconds (ns) and CPU clock ticks. Then the validation script collects the latency data. It compares the maximum latency measurement with the latency requirement specified in the script’s
    --latency_us
    command-line option. The option is a method of validating the latency value in the requirements file, in that users of the script must make sure they enter the same value in both the script and the requirements file. Code snippet from validation script:
    return process_workload_output(OUTPUT_FILE_NS, OUTPUT_FILE_TICKS, args)
  3. The workload validation script returns 0 if the maximum latency measurement is equal to or lower than the latency requirement, indicating validation was successful. Returns non-zero otherwise. The data streams optimizer tool processes the return code accordingly.
    The script also prints output. The output does not affect the behavior of the tool; the tool simply displays it.

Samples

For a working example of a workload validation script, see MRL Sample.

Product and Performance Information

1

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.