You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
**DecisionFocusedLearningBenchmarks.jl** provides a collection of benchmark problems for evaluating decision-focused learning algorithms. The package offers:
36
36
37
-
-**Standardized benchmark problems** spanning diverse application domains
38
-
-**Common interfaces** for creating datasets, statistical models, and optimization algorithms
39
-
-**Ready-to-use DFL policies**compatible with [InferOpt.jl](https://github.com/JuliaDecisionFocusedLearning/InferOpt.jl) and the whole [JuliaDecisionFocusedLearning](https://github.com/JuliaDecisionFocusedLearning) ecosystem
40
-
-**Evaluation tools** for comparing algorithm performance
37
+
-**Collection of benchmark problems** spanning diverse applications
38
+
-**Common tools** for creating datasets, statistical models, and optimization algorithms
39
+
-**Generic interface**for building custom benchmarks
40
+
-Compatibility with [InferOpt.jl](https://github.com/JuliaDecisionFocusedLearning/InferOpt.jl) and the whole [JuliaDecisionFocusedLearning](https://github.com/JuliaDecisionFocusedLearning) ecosystem
This guide covers everything you need to work with existing benchmarks in DecisionFocusedLearningBenchmarks.jl: generating datasets, assembling DFL pipeline components, applying algorithms, and evaluating results.
4
+
5
+
---
6
+
7
+
## What is a benchmark?
8
+
9
+
A benchmark bundles a problem family (an instance generator, a combinatorial solver, and a statistical model architecture) into a single object. It provides everything needed to run a Decision-Focused Learning experiment out of the box, without having to create each component from scratch.
10
+
Three abstract types cover the main settings:
11
+
-**`AbstractBenchmark`**: static problems (one instance, one decision)
12
+
-**`AbstractStochasticBenchmark{exogenous}`**: stochastic problems (type parameter indicates whether uncertainty is exogenous)
-**`generate_statistical_model`**: returns an untrained neural network that maps input features `x` to cost parameters `θ`.
31
+
-**`generate_maximizer`**: returns a callable `(θ; context...) -> y` that solves the combinatorial problem given cost parameters.
32
+
-**`generate_dataset`**: returns labeled training data as a `Vector{DataSample}`.
33
+
34
+
At inference time these two pieces compose naturally as an end-to-end policy:
35
+
36
+
```julia
37
+
θ =model(sample.x) # predict cost parameters
38
+
y =maximizer(θ; sample.context...) # solve the optimization problem
39
+
```
6
40
7
41
---
8
42
@@ -18,8 +52,7 @@ All data in the package is represented as [`DataSample`](@ref) objects.
18
52
|`context`|`NamedTuple`| Solver kwargs spread into `maximizer(θ; sample.context...)`|
19
53
|`extra`|`NamedTuple`| Non-solver data (scenario, reward, step, …), never passed to the solver |
20
54
21
-
Not all fields are populated in every sample. For convenience, named entries inside
22
-
`context` and `extra` can be accessed directly on the sample via property forwarding:
55
+
Not all fields are populated in every sample, depending on the setting. For convenience, named entries inside `context` and `extra` can be accessed directly on the sample via property forwarding:
23
56
24
57
```julia
25
58
sample.instance # looks up :instance in context first, then in extra
@@ -28,12 +61,11 @@ sample.scenario # looks up :scenario in context first, then in extra
28
61
29
62
---
30
63
31
-
## Generating datasets for training
64
+
## Benchmark type specifics
32
65
33
66
### Static benchmarks
34
67
35
-
For static benchmarks (`<:AbstractBenchmark`) the framework already computes the
36
-
ground-truth label `y`:
68
+
For static benchmarks (`<:AbstractBenchmark`), `generate_dataset` may compute a default ground-truth label `y` if the benchmark implements it:
37
69
38
70
```julia
39
71
bench =ArgmaxBenchmark()
@@ -43,15 +75,13 @@ dataset = generate_dataset(bench, 100; seed=0) # Vector{DataSample} with x, y,
43
75
You can override the labels by providing a `target_policy`:
For `AbstractStochasticBenchmark{true}` benchmarks the default call returns
54
-
*unlabeled* samples, each sample carries one scenario in `sample.extra.scenario`:
84
+
For `AbstractStochasticBenchmark{true}` benchmarks the default call returns *unlabeled* samples, each sample carries one scenario in `sample.extra.scenario`:
55
85
56
86
```julia
57
87
bench =StochasticVehicleSchedulingBenchmark()
@@ -85,20 +115,22 @@ Dynamic benchmarks use a two-step workflow:
85
115
```julia
86
116
bench =DynamicVehicleSchedulingBenchmark()
87
117
88
-
# Step 1 — create environments (reusable across experiments)
118
+
# Step 1: create environments (reusable across experiments)
89
119
envs =generate_environments(bench, 10; seed=0)
90
120
91
-
# Step 2 — roll out a policy to collect training trajectories
121
+
# Step 2: roll out a policy to collect training trajectories
92
122
policy =generate_baseline_policies(bench)[1] # e.g. lazy policy
In this tutorial, we showcase DecisionFocusedLearningBenchmarks.jl capabilities on one of its main benchmarks: the Warcraft benchmark.
8
+
This benchmark problem is a simple path-finding problem where the goal is to find the shortest path between the top left and bottom right corners of a given image map.
9
+
The map is represented as a 2D image representing a 12x12 grid, each cell having an unknown travel cost depending on the terrain type.
10
+
11
+
First, let's load the package and create a benchmark object as follows:
12
+
13
+
````@example warcraft_tutorial
14
+
using DecisionFocusedLearningBenchmarks
15
+
b = WarcraftBenchmark()
16
+
````
17
+
18
+
## Dataset generation
19
+
20
+
These benchmark objects behave as generators that can generate various needed elements in order to build an algorithm to tackle the problem.
21
+
First of all, all benchmarks are capable of generating datasets as needed, using the [`generate_dataset`](@ref) method.
22
+
This method takes as input the benchmark object for which the dataset is to be generated, and a second argument specifying the number of samples to generate:
23
+
24
+
````@example warcraft_tutorial
25
+
dataset = generate_dataset(b, 50);
26
+
nothing #hide
27
+
````
28
+
29
+
We obtain a vector of [`DataSample`](@ref) objects, containing all needed data for the problem.
30
+
Subdatasets can be created through regular slicing:
And getting an individual sample will return a [`DataSample`](@ref) with four fields: `x`, `info`, `θ`, and `y`:
37
+
38
+
````@example warcraft_tutorial
39
+
sample = test_dataset[1]
40
+
````
41
+
42
+
`x` correspond to the input features, i.e. the input image (3D array) in the Warcraft benchmark case:
43
+
44
+
````@example warcraft_tutorial
45
+
x = sample.x
46
+
````
47
+
48
+
`θ` correspond to the true unknown terrain weights. We use the opposite of the true weights in order to formulate the optimization problem as a maximization problem:
49
+
50
+
````@example warcraft_tutorial
51
+
θ_true = sample.θ
52
+
````
53
+
54
+
`y` correspond to the optimal shortest path, encoded as a binary matrix:
55
+
56
+
````@example warcraft_tutorial
57
+
y_true = sample.y
58
+
````
59
+
60
+
`context` is not used in this benchmark (no solver kwargs needed), so it is empty:
61
+
62
+
````@example warcraft_tutorial
63
+
isempty(sample.context)
64
+
````
65
+
66
+
For some benchmarks, we provide the following plotting method [`plot_data`](@ref) to visualize the data:
67
+
68
+
````@example warcraft_tutorial
69
+
plot_data(b, sample)
70
+
````
71
+
72
+
We can see here the terrain image, the true terrain weights, and the true shortest path avoiding the high cost cells.
73
+
74
+
## Building a pipeline
75
+
76
+
DecisionFocusedLearningBenchmarks also provides methods to build an hybrid machine learning and combinatorial optimization pipeline for the benchmark.
77
+
First, the [`generate_statistical_model`](@ref) method generates a machine learning predictor to predict cell weights from the input image:
78
+
79
+
````@example warcraft_tutorial
80
+
model = generate_statistical_model(b)
81
+
````
82
+
83
+
In the case of the Warcraft benchmark, the model is a convolutional neural network built using the Flux.jl package.
84
+
85
+
````@example warcraft_tutorial
86
+
θ = model(x)
87
+
````
88
+
89
+
Note that the model is not trained yet, and its parameters are randomly initialized.
90
+
91
+
Finally, the [`generate_maximizer`](@ref) method can be used to generate a combinatorial optimization algorithm that takes the predicted cell weights as input and returns the corresponding shortest path:
92
+
93
+
````@example warcraft_tutorial
94
+
maximizer = generate_maximizer(b; dijkstra=true)
95
+
````
96
+
97
+
In the case o fthe Warcraft benchmark, the method has an additional keyword argument to chose the algorithm to use: Dijkstra's algorithm or Bellman-Ford algorithm.
98
+
99
+
````@example warcraft_tutorial
100
+
y = maximizer(θ)
101
+
````
102
+
103
+
As we can see, currently the pipeline predicts random noise as cell weights, and therefore the maximizer returns a straight line path.
104
+
105
+
````@example warcraft_tutorial
106
+
plot_data(b, DataSample(; x, θ, y))
107
+
````
108
+
109
+
We can evaluate the current pipeline performance using the optimality gap metric:
0 commit comments