flexs.evaluate¶
A small set of evaluation metrics to benchmark explorers.
-
flexs.evaluate.adaptivity(landscape, make_explorer, num_rounds=[1, 10, 100], total_ground_truth_measurements=1000, total_model_queries=10000)[source]¶ For a fixed total budget of ground truth measurements and model queries, run with different numbers of rounds.
- Parameters
landscape (
Landscape) – Ground truth fitness landscape.make_explorer (
Callable[[int,int,int],Explorer]) – A function that takes in a number of rounds, a sequences_batch_size and a model_queries_per_batch and returns an explorer.num_rounds (
List[int]) – A list of number of rounds to run the explorer with.total_ground_truth_measurements (
int) – Total number of ground truth measurements across all rounds (sequences_batch_size * rounds).total_model_queries (
int) – Total number of model queries across all rounds (model_queries_per_round * rounds).
-
flexs.evaluate.efficiency(landscape, make_explorer, budgets=[(100, 500), (100, 5000), (1000, 5000), (1000, 10000)])[source]¶ Evaluate explorer outputs as a function of the number of allowed ground truth measurements and model queries per round.
- Parameters
landscape (
Landscape) – Ground truth fitness landscape.make_explorer (
Callable[[int,int],Explorer]) – A function that takes in a sequences_batch_size and a model_queries_per_batch and returns an explorer.budgets (
List[Tuple[int,int]]) – A list of tuples (sequences_batch_size, model_queries_per_batch).