MiniZinc - Challenge 2018 Results

MiniZinc Challenge 2018 Results

CP 2018 presentation slides of the MiniZinc Challenge 2018 will be available here after the announcement. (results). Note that the slides do not contain the amendments of the results.

Entrants

The entrants for this year (with their descriptions, when provided):

Choco 4 (description). A Java FD solver.
Concrete (description). A CP system written in Scala.
HaifaCSP (description). A C++ FD solver using SAT solving algorithms.
iZplus (description).
JaCoP (description). A Java FD solver.
OR-Tools (description).
OscaR/CBLS (description). A constraint-based local search solver written in Scala.
Picat SAT (description).
SICStus Prolog (description). A Prolog development environment with a FD constraint programming module.
sunny-cp^— (description). A variant of sunny-cp only using Choco, Gecode, HaifaCSP, JaCoP, iZplus, MinisatID, Mistral, Opturion CPX, OR-Tools, Picat.
Yuck (description). A local search solver written in Scala.

In addition, the challenge organisers entered the following FlatZinc and MiniZinc implementations:

Chuffed (description). A C++ FD solver using lazy clause generation.
Gecode (description). A C++ FD solver.
MZN/Cbc. Translates to MILP (description), uses Cbc version 2.9.
MZN/CPLEX. Translates to MILP (description), uses IBM ILOG CPLEX Optimizer version 12.6.0.0.
sunny-cp (description). A multi-threaded CP portfolio solver using 13 different CP and MIP solvers incl. Chuffed, G12 solvers, Gecode.

As per the challenge rules, these entries are not eligible for prizes, but do modify the scoring results.

Furthermore, entries in the FD search category (Gecode, JaCoP, SICStus Prolog) were automatically included in the free search category, while entries in the free search category (Chuffed, Concrete, HaifaCSP, OscaR/CBLS, Yuck and promoted FD entries except Gecode) were automatically included in the parallel search category. Lastly, all entries in the parallel search category and promoted entries into that category were automatically included in the open search category.

Amendments of Results - 20 September 2018

Unfortunately, there was an issue with the re-implemented solution checker this year. The checker considered redundant and symmetry breaking constraints in the model for checking the correctness of the solvers' solutions leading to false positives. The issue has been addressed now and all solutions of each solver were re-checked leading to following changes.

The solvers OscaR/CBLS and Yuck pass all previously-failed tests (24 instances each).
The solvers MZN/Cbc and MZN/CPLEX pass three previously-failed instances each.
The solver sunny-cp^— fails one previously-passed instance.

Due to these changes, there were changes in the rankings, which also affected the Bronze medal in the free search category. The solver Choco 4 overtook the solver HaifaCSP in this category. The Judges were informed and decided to jointly award the Bronze medal to both solvers, because the solvers' scores differ only by a small margin and the issue with the checker was out of their control. We deeply regret and apologise for our mistake and thank to Gustav Björdal and Michael Marte, who independently reported us about this mistake.

Summary of Results

The results for the MiniZinc Challenge 2018 are

Category	Gold	Silver	Bronze
Fixed	OR-Tools	JaCoP	Choco 4
Free	OR-Tools	Picat SAT	Choco 4 + HaifaCSP
Parallel	OR-Tools	Choco 4	Picat SAT
Open	OR-Tools	sunny-cp—	Choco 4
Local Search	iZplus	Yuck	OscaR/CBLS

Description of Results

All times are given in milliseconds.

A score of 0.0 indicates a worse answer in quality (worse objective, no proof of optimality, or no answer for satisfaction problems), 1.0 a better solution in quality. When the quality is the same, the 1.0 purse is split with respect to time used.

If a promoted entry does not recognize an option (or states that it is just ignored), times and solutions from the previous category are used for scoring. The suffixes -fd, -free, -par or -open (for the parallel portfolio solver entered) at the end of the solver names indicate which configuration the solvers were run with.

mzn2fzn was run with the same time and memory limits as the solvers.

In the Status column:

S indicates that a solution was found,
C indicates that the search was complete,
ERR indicates an incorrect answer or the solver aborted,
ERR indicates that flattening aborted (time-out or out-of-memory),
UNK indicates that no answer was returned in the time limit.

Download all problems

All problems are available in a zipped tar-ball here.

Selection:

Select a list of solvers and benchmarks and click on “Compute Results” to score the solvers against each other on the selected benchmarks. The entrants for each of the fd search, free search and parallel search categories can be selected with the corresponding buttons.

Solver selection:	Problem selection:

Plot settings:
Note that plotting takes time. Create plots for each instance Time axis scale: Default Linear Log Plot lines shape: Steps Lines
Area scoring settings:
Normalise

Summary:

Total per problem:

Solver	Score	Score Incomplete	Score Area

Problem	Solver	Score	Score Incomplete	Score Area

Individual results:

Problem	Instance	Solver	Status	Time	Objective	Score	Score Incomplete	Score Area

Objective plots for each instance:

Problem	Instance	Plot

Score development over time:

Global constraint per model

The following table lists the global constraints used by each model in this year's challenge. In addition, the columns RC and SBC, respectively, indicate whether the model contains redundant or/and symmetry breaking constraints.

Problem	Type	Kind	RC	SBC	MiniZinc Globals
cargo	real	min			cumulative, diffn
concert-hall-cap	combi	max		✓	all_different_except_0, value_precede_chain
elitserien	real	min			all_different, global_cardinality_closed, inverse, member, regular
gfd-schedule2	real	min		✓	at_most, cumulative, nvalue
largescheduling	combi	min	✓		cumulative
mapping	real	min			bin_packing_load, count, maximum, network_flow_cost
neighbours	puzzle	max	✓	✓	lex_lesseq
on-call-rostering	combi	min			count
oocsp_racks	real	sat			global_cardinality, increasing
opt-cryptanalysis	real	min			table
proteindesign12	real	min			table
racp	combi	min	✓		cumulative
rotating-workforce	real	sat	✓	✓	global_cardinality_low_up, regular
seat-moving	combi	min	✓		all_different, all_different_except_0
soccer-computational	real	sat			all_different
steiner-tree	combi	min	✓
team-assignment	real	max	✓	✓	all_different, bin_packing, bin_packing_load
test-scheduling	real	min	✓	✓	cumulative, disjunctive, value_precede_chain
train	real	min
vrplc	real	min		✓	circuit, cumulative

The files on this page are for MiniZinc version 2.1.7.