Comparing Regression Data

1Automated Commit Testing

The process begins by fetching tardis commit hashes from the tardis repository. The commits can be retrieved through two different methods: fetching the last n tardis commits or providing a custom commit list. Once the commits are obtained through either approach, they are then processed for the next step.

2Environment Management

The system follows a structured workflow to create TARDIS environments for each commit to ensure regression data reproducibility. It provides various options and provides detailed configurable log statements at each step.

The workflow by default automatically fetches the environment file for each commit and creates a fresh environment for it to process the regression data in. TARDIS uses conda lockfiles which have precise dependency versions for recreating the environment.

Once the environment is installed, the workflow also fetches the optional dependencies from pyproject.toml automatically(TARDIS has some pip based dependencies as optional dependency for visualisation modules). After installing the dependencies, the workflow proceeds to run tests for that commit.

The workflow also allows to reuse environments if they have already been created in previous runs.

At any step, if the workflow fails to process any step, for e.g., installing optional dependencies, it will default to the default environment but will continue to process commits. The system warns the user that such failure has occurred, since failed installations and the regression data produced from such commits won't be as reliable from a scientific perspective.

3Testing

For each commit, the system executes pytest to run both continuum and non-continuum test suites. Once testing is complete, the generated regression data from these test runs is committed to the repository, creating a new commit that captures the test results.

4Getting regression Data commits

There are two ways to proceed after this step. The first option is to use falsely generated regression commits that are created programmatically from the tardis commits obtained in the previous step. The second option is to fetch regression data commits directly from git, which can be done by either retrieving the last n commits or by providing custom commit hashes.

5Comparison of Regression Data commits

Once the regression data commits have been finalised, it's time to move to the comparison step. If you wish, you can also preview the commits with details about how they were generated before moving forward.

The workflow compares two commits at a time, running git diff on each pair. For more comprehensive testing, the workflow automatically copies over contents from each item into temporary directories. The current code only allows git based diffs and command line based diffs but is modular so that new comparison methods can be added easily.

There are filtering options for .H5 and .npy files, too in case you want to just view certain types of files in the comparison.

The comparison matrix creates a table where each column is compared between two consecutive commits, and each cell under those columns is either modified(M), unchanged(•), not-present(-), deleted(D) or added(A).