Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 17 additions & 7 deletions docs/src/custom_benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,14 @@ problems to the benchmark suite or integrate their own domains.

```
AbstractBenchmark
└── AbstractStochasticBenchmark{exogenous}
└── AbstractDynamicBenchmark{exogenous}
├── AbstractStaticBenchmark
├── AbstractStochasticBenchmark{exogenous}
└── AbstractDynamicBenchmark{exogenous}
```

| Type | Use case |
|------|----------|
| `AbstractBenchmark` | Static, single-stage optimization (e.g. shortest path, portfolio) |
| `AbstractStaticBenchmark` | Static, single-stage optimization (e.g. shortest path, portfolio) |
| `AbstractStochasticBenchmark{true}` | Single-stage with exogenous uncertainty (scenarios drawn independently of decisions) |
| `AbstractStochasticBenchmark{false}` | Single-stage with endogenous uncertainty |
| `AbstractDynamicBenchmark{true}` | Multi-stage sequential decisions with exogenous uncertainty |
Expand All @@ -41,7 +42,7 @@ repeatedly and applies `target_policy` to each result.

---

## `AbstractBenchmark`: required methods
## `AbstractStaticBenchmark`: required methods

### Data generation (choose one strategy)

Expand Down Expand Up @@ -70,7 +71,7 @@ generate_maximizer(bench::MyBenchmark)

```julia
generate_baseline_policies(bench::MyBenchmark) -> collection of callables
is_minimization_problem(bench::MyBenchmark) -> Bool # default: false (maximization)
is_minimization_problem(bench::MyBenchmark) -> Bool # default: true (minimization)
objective_value(bench::MyBenchmark, sample::DataSample, y) -> Real
compute_gap(bench::MyBenchmark, dataset, model, maximizer) -> Float64
has_visualization(bench::MyBenchmark) -> Bool # default: false; return true when plot methods are implemented/available
Expand All @@ -80,7 +81,7 @@ plot_solution(bench::MyBenchmark, sample::DataSample; kwargs...)

---

## `AbstractStochasticBenchmark{true}`: additional methods
## `AbstractStochasticBenchmark{true}`

For stochastic benchmarks with exogenous uncertainty, implement:

Expand Down Expand Up @@ -113,7 +114,7 @@ DataSample(; x=features, y=nothing,

---

## `AbstractDynamicBenchmark`: additional methods
## `AbstractDynamicBenchmark`

Dynamic benchmarks extend stochastic ones with an environment-based rollout interface.

Expand Down Expand Up @@ -148,6 +149,15 @@ generate_baseline_policies(bench::MyDynamicBenchmark)
# Each callable performs a full episode rollout and returns the trajectory.
```

### Anticipative solver (optional)

```julia
generate_anticipative_solver(bench::MyDynamicBenchmark)
# Returns a callable: (env; reset_env=true, kwargs...) -> Vector{DataSample}
# reset_env=true → reset environment before solving
# reset_env=false → solve from current state
```

### Optional visualization methods

```julia
Expand Down
2 changes: 1 addition & 1 deletion docs/src/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Where:

The package organizes benchmarks into three main categories based on their problem structure:

### Static Benchmarks (`AbstractBenchmark`)
### Static Benchmarks (`AbstractStaticBenchmark`)
Single-stage optimization problems with no randomness involved:
- [`ArgmaxBenchmark`](@ref): argmax toy problem
- [`Argmax2DBenchmark`](@ref): 2D argmax toy problem
Expand Down
7 changes: 4 additions & 3 deletions docs/src/using_benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This guide covers everything you need to work with existing benchmarks in Decisi

A benchmark bundles a problem family (an instance generator, a combinatorial solver, and a statistical model architecture) into a single object. It provides everything needed to run a Decision-Focused Learning experiment out of the box, without having to create each component from scratch.
Three abstract types cover the main settings:
- **`AbstractBenchmark`**: static problems (one instance, one decision)
- **`AbstractStaticBenchmark`**: static problems (one instance, one decision)
- **`AbstractStochasticBenchmark{exogenous}`**: stochastic problems (type parameter indicates whether uncertainty is exogenous)
- **`AbstractDynamicBenchmark{exogenous}`**: sequential / multi-stage problems

Expand Down Expand Up @@ -65,7 +65,7 @@ sample.scenario # looks up :scenario in context first, then in extra

### Static benchmarks

For static benchmarks (`<:AbstractBenchmark`), `generate_dataset` may compute a default ground-truth label `y` if the benchmark implements it:
For static benchmarks (`<:AbstractStaticBenchmark`), `generate_dataset` may compute a default ground-truth label `y` if the benchmark implements it:

```julia
bench = ArgmaxBenchmark()
Expand Down Expand Up @@ -150,7 +150,8 @@ dataset = generate_dataset(bench, 50; rng=rng)
gap = compute_gap(bench, dataset, model, maximizer)
```

# Objective value for a single decision
Objective value for a single decision:

```julia
obj = objective_value(bench, sample, y)
```
Expand Down
2 changes: 1 addition & 1 deletion src/Argmax/Argmax.jl
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Basic benchmark problem with an argmax as the CO algorithm.
# Fields
$TYPEDFIELDS
"""
struct ArgmaxBenchmark{E} <: AbstractBenchmark
struct ArgmaxBenchmark{E} <: AbstractStaticBenchmark
"instances dimension, total number of classes"
instance_dim::Int
"number of features"
Expand Down
2 changes: 1 addition & 1 deletion src/Argmax2D/Argmax2D.jl
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Argmax becnhmark on a 2d polytope.
# Fields
$TYPEDFIELDS
"""
struct Argmax2DBenchmark{E,R} <: AbstractBenchmark
struct Argmax2DBenchmark{E,R} <: AbstractStaticBenchmark
"number of features"
nb_features::Int
"true mapping between features and costs"
Expand Down
11 changes: 0 additions & 11 deletions src/DynamicAssortment/DynamicAssortment.jl
Original file line number Diff line number Diff line change
Expand Up @@ -78,17 +78,6 @@ include("policies.jl")
"""
$TYPEDSIGNATURES

Outputs a data sample containing an [`Instance`](@ref).
"""
function Utils.generate_sample(
b::DynamicAssortmentBenchmark, rng::AbstractRNG=MersenneTwister(0)
)
return DataSample(; instance=Instance(b, rng))
end

"""
$TYPEDSIGNATURES

Generates a statistical model for the dynamic assortment benchmark.
The model is a small neural network with one hidden layer of size 5 and no activation function.
"""
Expand Down
2 changes: 1 addition & 1 deletion src/FixedSizeShortestPath/FixedSizeShortestPath.jl
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Data is generated using the process described in: <https://arxiv.org/abs/2307.13
# Fields
$TYPEDFIELDS
"""
struct FixedSizeShortestPathBenchmark <: AbstractBenchmark
struct FixedSizeShortestPathBenchmark <: AbstractStaticBenchmark
"grid graph instance"
graph::SimpleDiGraph{Int64}
"grid size of graphs"
Expand Down
2 changes: 1 addition & 1 deletion src/PortfolioOptimization/PortfolioOptimization.jl
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Data is generated using the process described in: <https://arxiv.org/abs/2307.13
# Fields
$TYPEDFIELDS
"""
struct PortfolioOptimizationBenchmark <: AbstractBenchmark
struct PortfolioOptimizationBenchmark <: AbstractStaticBenchmark
"number of assets"
d::Int
"size of feature vectors"
Expand Down
2 changes: 1 addition & 1 deletion src/Ranking/Ranking.jl
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Basic benchmark problem with ranking as the CO algorithm.
# Fields
$TYPEDFIELDS
"""
struct RankingBenchmark{E} <: AbstractBenchmark
struct RankingBenchmark{E} <: AbstractStaticBenchmark
"instances dimension, total number of classes"
instance_dim::Int
"number of features"
Expand Down
2 changes: 1 addition & 1 deletion src/SubsetSelection/SubsetSelection.jl
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ without knowing their values, but only observing some features.
# Fields
$TYPEDFIELDS
"""
struct SubsetSelectionBenchmark{M} <: AbstractBenchmark
struct SubsetSelectionBenchmark{M} <: AbstractStaticBenchmark
"total number of items"
n::Int
"number of items to select"
Expand Down
8 changes: 6 additions & 2 deletions src/Utils/Utils.jl
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,10 @@ include("data_sample.jl")
include("maximizers.jl")
include("environment.jl")
include("policy.jl")
include("interface.jl")
include("interface/abstract_benchmark.jl")
include("interface/static_benchmark.jl")
include("interface/stochastic_benchmark.jl")
include("interface/dynamic_benchmark.jl")
include("grid_graph.jl")
include("misc.jl")
include("model_builders.jl")
Expand All @@ -26,7 +29,8 @@ export TopKMaximizer, one_hot_argmax

export AbstractEnvironment, get_seed, is_terminated, observe, reset!, step!

export AbstractBenchmark, AbstractStochasticBenchmark, AbstractDynamicBenchmark
export AbstractBenchmark,
AbstractStaticBenchmark, AbstractStochasticBenchmark, AbstractDynamicBenchmark
export ExogenousStochasticBenchmark,
EndogenousStochasticBenchmark, ExogenousDynamicBenchmark, EndogenousDynamicBenchmark
export generate_instance, generate_sample, generate_dataset
Expand Down
Loading
Loading