Motivation for relaunched competition
The main motivation for running the ILP 2016 competition is that there are many ILP systems, each with their own strengths and weaknesses. Systems have been tested on various datasets but, as most of these are not publically available and only highlight the strengths of some of the systems, there is not a standard set of benchmarks for all ILP systems.
The aim of this competition is to provide a set of benchmarks together with a framework to test each entered system in the same environment.
Due to a lack of entries, the original competition did not go ahead. At ILP 2016, we represented the competition and held a discussion on the possible reasons why people decided not to enter.
We came to the conclusion that possible reasons for people not entering were:
- Information about the competition was not accessible enough before entering.
- There was no incentive (other than pride) in entering the competition.
- People were concerned that their systems may not perform well in certain categories (e.g. their system is monotonic).
- People were not interested in a purely synthetic domain.
To address the first point, we have made much more information about the competition available on the website without the need to sign up. Users can now see information on the problem domain, and some example problems.
In order to incentivise entrants, we have decided to invite the systems with the best average score (in both the probabilistic, and non-probabilistic tracks) to submit a full length article to the 2016 ILP special issue of the MLJ.
During ILP we tried to reassure potential entrants that there is no need to be concerned about entering systems that are likely to only perform well in some of the categories. In fact, we would be surprised if there is a system that performs well in every category. While the winners will be the entrants whose average score is best across all categories, the results will be available in a format that makes it easy to analyse the relative strengths of each system (e.g. you might be able to claim "My system is best at problems that do not involve predicate invention").
There are several reasons for our choice of problem domain:
- Firstly, the competition was designed to test the flexibility of the systems on learning different kinds of hypotheses (which may or may not involve non-monotonicity, recursion, predicate invention or non-observational predicate learning). We therefore needed a single problem domain that could involve each of these features.
- Secondly, we would like to test the systems on unseen problems. The entrants will be able to tailor their systems to the problem domain (by using the example problems), but will not, for example, be able to make their language biases too specific, as the target hypotheses will be unknown before the testing phase.