The competition is currently in the development phase, where we invite you to develop and test your methods on the refernece data for which we release the HMC samples. To walk you through the process of loading the data, training the model, and making a submission we provide the following Google Colab’s:
In general, we recommend trying colab for the competition, as it provides convenient environments and free GPUs. Here, we will discuss the submission process without language- and framework-specific details.
Accessing the data
We provide the data as
.csv files via Google cloud storage. You can download the data via gsutil as
gsutil -m cp -r gs://neurips2021_bdl_competition/*.csv .
- CIFAR-10 train features; CIFAR-10 train labels; CIFAR-10 test features; CIFAR-10 test labels
- IMDB train features; IMDB train labels; IMDB test features; IMDB test labels
For the development phase, we will be evaluating predictions on CIFAR-10 and IMDB datasets in our submission system, but we will also release additional datasets and the corresponding HMC checkpoints for reference. For an up-to-date list of available model-dataset pairs see the resources tab.
.csv files with features each contain
n_data rows and
n_features columns, where
n_data is the number of datapoints and
n_features is the number of features.
The files with labels contain
n_data rows and 1 column.
See the Getting started in JAX colab for an example of creating a dataloader from the provided
We provide reference implementations of the models used for the competition in JAX and PyTorch;
resnet20_frn_swish is used on CIFAR-10 and
cnn_lstm is used on IMDB:
We will update the model files to include new architectures over time. If you intend to use a different language or framework, you will need to re-implement the models in your framework of choice.
We manage submissions via a CodaLab competition.
The submissions consist of a
zip file containing
Each of these files contains
n_input rows and
n_class columns, where
n_input is the number of test datapoints and
n_class is the number of classes.
The value at row
j should be the predicted probability of class
j for the input number
i in the test set.
To create the
zip file for submission you can use e.g.
zip submission.zip cifar10_probs.csv imdb_probs.csv
To upload your submission, use the participate tab on CodaLab.