Evaluation options

mtt eval evaluates a trained model on a dataset described by an options file:

mtt eval model.pt eval.yaml

The systems and targets sections of eval.yaml are the same as for a dataset in the training options file (see Training YAML Reference). When targets are given, the errors of the model on them are reported. This page describes the options that change which errors are reported.

class metatrain.share.base_hypers.EvalDatasetDictHypers

systems: str | SystemsHypers

Path to the dataset file or a dictionary specifying the dataset.

targets: NotRequired[dict[str, Annotated[TargetHypers, AfterValidator(func=sanitize_target_hypers)] | str]]

A dictionary specifying the targets in the dataset.

extra_data: NotRequired[dict]

Additional data to include from the dataset.

indices: NotRequired[list[int] | str]

Explicit indices to select from the dataset.

Can be either a list of integers (e.g., [0, 1, 5, 10]) or a path to a text file containing one index per line. When specified, only the structures at these indices will be used from the dataset.

equivariance: NotRequired[EquivarianceHypers]

Also measure the O(3) equivariance error of the model.

Requires a targets section. Use equivariance: {} to enable it with the default settings. Every system is additionally evaluated on many rotated and inverted copies, so evaluation becomes substantially slower. The O(3)-averaged predictions and their variance over the orientations are written to the output file next to the predictions, as o3::average::<target> and o3::variance::<target>.

log_separate_blocks: NotRequired[bool] = False

Report metrics separately for each block of each target, instead of aggregated over the whole target.

Each per-block metric is labeled with the block’s key, e.g. for a spherical target decomposed by o3_lambda/o3_sigma/atom_type: "<target> (o3_lambda=1,o3_sigma=1,atom_type=6) RMSE". Matches the log_separate_blocks training hyperparameter.

Errors for each block

By default, the errors are aggregated over all the blocks of a target. To report them separately for each block, e.g. for each o3_lambda and o3_sigma of a spherical target, set log_separate_blocks:

systems: dataset.xyz
targets:
  mtt::dipole:
    type:
      spherical:
        irreps:
          - {o3_lambda: 1, o3_sigma: 1}
log_separate_blocks: true

Each error is then labeled with the key of its block, and the aggregated errors are not reported:

mtt::dipole (o3_lambda=1,o3_sigma=1) RMSE (per atom): 0.0123 | ...

Equivariance error

Models that are not equivariant by construction, i.e. whose predictions do not rotate exactly like the targets, e.g. PET, can learn equivariance only approximately. mtt eval can measure how far a model is from being equivariant, by also evaluating every system on many rotated and inverted copies, rotating the predictions back, and comparing them. Add an equivariance section (equivariance: {} uses the default settings), which requires a targets section:

systems: dataset.xyz
targets:
  energy:
    key: U0
equivariance:
  max_angular_momentum_grid: 6

For each target, the following errors are then reported, in the units of the target:

• equivariance RMSE: the spread of the back-rotated predictions over the orientations, which is zero for an equivariant model;

• O3-averaged RMSE and O3-averaged MAE: the errors of the predictions averaged over all orientations, i.e. of the closest equivariant model;

• oriented RMSE: sqrt(equivariance RMSE² + O3-averaged RMSE²), the RMSE the model would have on randomly oriented copies of the systems.

energy equivariance RMSE (per atom): 0.28 meV | energy O3-averaged RMSE (per atom): 0.63 meV | energy oriented RMSE (per atom): 0.69 meV | ...

With log_separate_blocks, the equivariance error is also reported for each block. For scalar and Cartesian targets, these blocks are labeled by the angular momentum components (o3_lambda, o3_sigma) of the target, rather than by its own keys.

The orientations are sampled on a quadrature grid, whose degree is set by max_angular_momentum_grid. The default grid is exact only for models that are equivariant by construction. mtt eval therefore re-measures the error of the first batch on a finer grid, and warns if the two differ by more than 1%; in that case, increase max_angular_momentum_grid until the error stops changing. The cost of the evaluation grows steeply with the degree of the grid.

The rotated copies of each system are evaluated in batches of mtt eval’s batch size (-b), which defaults to 16 instead of 1 when any dataset measures the equivariance error.

The equivariance error of gradients (e.g. forces) is not measured: a warning is printed and only the target values are considered.

The O(3)-averaged predictions and their variance over the orientations are written to the output file of mtt eval, next to the predictions, under o3::average::<target> and o3::variance::<target> (e.g. o3::average::energy and o3::variance::energy). The variance of a rank-2 Cartesian target is split into one block per angular momentum, so it can only be written to a .zip or .mts output.

class metatrain.share.base_hypers.EquivarianceHypers

max_angular_momentum_grid: NotRequired[int | None] = None

Degree of the O(3) quadrature used to measure the equivariance error.

When None, metatomic picks a default from the maximum angular momentum of the targets, which is exact only for models that are equivariant by construction. The cost grows steeply with this value.