Create runner for CBBO

[Deathn0t]

We will start by defining a runner.py script in each benchmark folder to execute benchmarks.

We will use TOML to define the configuration of a benchmark run.

title = "CBBO Benchmark"

[search]
max_evals = 100
num_replications = 2

[search.method.cbo]
package = "deephyper.hpo"
name = "CBO"

[search.method.tpe]
package = "deephyper_benchmark.hpo.optuna"
name = "OptunaSearch"


[search.method.random]
package = "deephyper.hpo"
name = "RandomSearch"

[evaluator]
method = "thread"
num_workers = 1

[benchmark.ackley]
package = "deephyper_benchmark.benchmarks.cbbo"
name = "AckleyBenchmark"

[Deathn0t]

An initial runner was created in https://github.com/deephyper/benchmark/blob/runner/src/deephyper_benchmark/benchmarks/cbbo/runner.py

To run it, move to examples/cbbo_runner then run the command:

python -m deephyper_benchmark.benchmarks.cbbo.runner --config cbbo.toml

This will create a directory structure such as:

.
├── cbbo.toml
└── runs
    └── ackley
        ├── cbo
        │   ├── 168c3143-794c-4d68-a30c-340ce47ccfb7
        │   │   ├── context.yaml
        │   │   └── results.csv
        │   └── 86dae083-ca6d-4dfc-bdfb-6de7b936ac7c
        │       ├── context.yaml
        │       └── results.csv
        ├── random
        │   ├── 987a3d4d-b43d-4c2e-ba0b-08ce60a389d6
        │   │   ├── context.yaml
        │   │   └── results.csv
        │   └── da83e868-0147-4c58-b110-a4c39ac510ab
        │       ├── context.yaml
        │       └── results.csv
        └── tpe
            ├── 9a0c667d-ca22-4a19-90b0-2e6fdb2620ab
            │   ├── context.yaml
            │   └── results.csv
            └── dd586a6c-c984-4b52-8776-2dced49fa41a
                ├── context.yaml
                └── results.csv

[Deathn0t]

Example printed outputs:

Running: CBBO Benchmark
Starting benchmark: ackley
Starting replica: ackley/cbo/0c9f10f3-643b-44c6-8a6a-33e80ecaeaae'
ackley/cbo/0c9f10f3-643b-44c6-8a6a-33e80ecaeaae:  50%|███████████████████████████████████▌                                   | 250/500 [01:03<01:36,  2.60it/s, failures=0, objective=-0.034]

[Deathn0t]

Following the same logic, I added a plotter:

python -m deephyper_benchmark.benchmarks.cbbo.plotter --config cbbo.toml

This command is to run from the same location as the runner.

It will creates the plots folder such as:

.
├── cbbo.toml
├── plots
│   └── ackley
│       └── average_simple_regret.png
└── runs
    ├── ackley
    │   ├── cbo
    │   │   ├── 094e8e08-8b2e-4e14-b6dd-ed7fa83f84e4
    │   │   │   ├── context.yaml
    │   │   │   └── results.csv

and create figures such as:

[wigging]

There is no need for this:

python -m deephyper_benchmark.benchmarks.cbbo.plotter --config cbbo.toml

Just provide an entry point for the command line interface (CLI) and run it like this:

deephyper-benchmark cbbo run

Or the plotter like this:

deephyper-benchmark cbbo plot

[Deathn0t]

@wigging can you add the entrypoints/parsers?

[wigging]

Is the cbbo runner done? Once it's done you can merge it with the main branch. Then I can start another branch and do a merge request for the command line interface.

[Deathn0t]

@wigging merged in #36, you can proceed