{
"cells": [
{
"cell_type": "markdown",
"id": "bf2d88b2-448d-4930-8302-2786c9c0057b",
"metadata": {},
"source": [
"# Evaluators\n",
"\n",
"Coba Evaluators evaluate the performance of Learners in Environments. They have the following interface.\n",
"\n",
"```python\n",
"class Evaluator:\n",
" def evaluate(self, environment: Environment, learner: Learner) -> Iterable[dict]\n",
" ...\n",
"```\n",
"\n",
"Environments and Learners are described in more detail in [Environments.ipynb](Environments.ipynb) and [Learners.ipynb](Learners.ipynb). In this notebook we...\n",
"\n",
"+ ...show how to create your own custom evaluators\n",
"+ ...describe the Evaluators provided by Coba out-of-the-box\n"
]
},
{
"cell_type": "markdown",
"id": "0ca048d1-4376-4077-be5c-b0c5bbbd9c92",
"metadata": {},
"source": [
"## Custom Evaluator\n",
"\n",
"Any class that conforms to the evaluator interface above can be used as an Evalutor in an Experiment. For example,"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "f805573a-d6d3-429b-be2f-105593fa30bd",
"metadata": {},
"outputs": [],
"source": [
"class MyEvaluator:\n",
" def evaluate(self, environment, learner):\n",
" yield {'a':1, 'b':2}\n",
" yield {'a':2, 'b':3}"
]
},
{
"cell_type": "markdown",
"id": "c8409cae-cb4d-423c-b9ba-2f5cd61e0929",
"metadata": {},
"source": [
"We can then pass our custom Evaluator to Experiment in order to generate results."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "a6a45114-da10-4fb8-8b9c-c98e061bddb1",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2024-02-02 14:37:33 -- Experiment Started\n",
"2024-02-02 14:37:33 -- Recording Learner 0 parameters... (0.0 seconds) (completed)\n",
"2024-02-02 14:37:33 -- Recording Evaluator 0 parameters... (0.0 seconds) (completed)\n",
"2024-02-02 14:37:33 -- Peeking at Environment 0... (0.0 seconds) (completed)\n",
"2024-02-02 14:37:33 -- Recording Environment 0 parameters... (0.0 seconds) (completed)\n",
"2024-02-02 14:37:33 -- Evaluating Learner 0 on Environment 0... (0.0 seconds) (completed)\n",
"2024-02-02 14:37:33 -- Experiment Finished\n"
]
}
],
"source": [
"import coba as cb\n",
"\n",
"env = cb.Environments.from_linear_synthetic(100)\n",
"lrn = cb.RandomLearner()\n",
"val = MyEvaluator()\n",
"\n",
"result = cb.Experiment(env,lrn,val).run()"
]
},
{
"cell_type": "markdown",
"id": "0bd8bbef-f51c-472a-a367-520e411405c7",
"metadata": {},
"source": [
"When we look at the result.evaluators we can see that MyEvaluator was used."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "7627bff8-31bf-40a7-b1d7-c057fc3c2a4f",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" evaluator_id | \n",
" eval_type | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 0 | \n",
" MyEvaluator | \n",
"
\n",
" \n",
"
\n",
"
\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" environment_id | \n",
" learner_id | \n",
" evaluator_id | \n",
" index | \n",
" a | \n",
" b | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 0 | \n",
" 0 | \n",
" 0 | \n",
" 1 | \n",
" 1 | \n",
" 2 | \n",
"
\n",
" \n",
" | 1 | \n",
" 0 | \n",
" 0 | \n",
" 0 | \n",
" 2 | \n",
" 2 | \n",
" 3 | \n",
"
\n",
" \n",
"
\n",
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"result.plot_learners(x='a',y='b')"
]
},
{
"cell_type": "markdown",
"id": "3ea52911-f41a-4cda-8c8d-5e9a1e1c8849",
"metadata": {},
"source": [
"If more than one Environment/Learner is passed to Experiment our custom Evalautor will be called for all combinations."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "19661bf1-2bfc-4472-b1d5-89bde23b4b19",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" environment_id | \n",
" learner_id | \n",
" evaluator_id | \n",
" index | \n",
" a | \n",
" b | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 0 | \n",
" 0 | \n",
" 0 | \n",
" 1 | \n",
" 1 | \n",
" 2 | \n",
"
\n",
" \n",
" | 1 | \n",
" 0 | \n",
" 0 | \n",
" 0 | \n",
" 2 | \n",
" 2 | \n",
" 3 | \n",
"
\n",
" \n",
" | 2 | \n",
" 0 | \n",
" 1 | \n",
" 0 | \n",
" 1 | \n",
" 1 | \n",
" 2 | \n",
"
\n",
" \n",
" | 3 | \n",
" 0 | \n",
" 1 | \n",
" 0 | \n",
" 2 | \n",
" 2 | \n",
" 3 | \n",
"
\n",
" \n",
" | 4 | \n",
" 1 | \n",
" 0 | \n",
" 0 | \n",
" 1 | \n",
" 1 | \n",
" 2 | \n",
"
\n",
" \n",
" | 5 | \n",
" 1 | \n",
" 0 | \n",
" 0 | \n",
" 2 | \n",
" 2 | \n",
" 3 | \n",
"
\n",
" \n",
" | 6 | \n",
" 1 | \n",
" 1 | \n",
" 0 | \n",
" 1 | \n",
" 1 | \n",
" 2 | \n",
"
\n",
" \n",
" | 7 | \n",
" 1 | \n",
" 1 | \n",
" 0 | \n",
" 2 | \n",
" 2 | \n",
" 3 | \n",
"
\n",
" \n",
"
\n",
"