Approximate Inference in Bayesian Deep Learning

Home

Getting Started

Submission System

Resources

FAQ

Twitter

Getting Started

The competition is currently in the development phase, where we invite you to develop and test your methods on the refernece data for which we release the HMC samples. To walk you through the process of loading the data, training the model, and making a submission we provide the following Google Colab’s:

In general, we recommend trying colab for the competition, as it provides convenient environments and free GPUs. Here, we will discuss the submission process without language- and framework-specific details.


Accessing the data

We provide the data as .csv files via Google cloud storage. You can download the data via gsutil as

gsutil -m cp -r gs://neurips2021_bdl_competition/*.csv .

or manually:

For the development phase, we will be evaluating predictions on CIFAR-10 and IMDB datasets in our submission system, but we will also release additional datasets and the corresponding HMC checkpoints for reference. For an up-to-date list of available model-dataset pairs see the resources tab.

The .csv files with features each contain n_data rows and n_features columns, where n_data is the number of datapoints and n_features is the number of features. The files with labels contain n_data rows and 1 column. See the Getting started in JAX colab for an example of creating a dataloader from the provided .csv files.


Model architectures

We provide reference implementations of the models used for the competition in JAX and PyTorch; resnet20_frn_swish is used on CIFAR-10 and cnn_lstm is used on IMDB:

We will update the model files to include new architectures over time. If you intend to use a different language or framework, you will need to re-implement the models in your framework of choice.


Submissions

We manage submissions via a CodaLab competition. The submissions consist of a zip file containing cifar10_probs.csv and imdb_probs.csv files. Each of these files contains n_input rows and n_class columns, where n_input is the number of test datapoints and n_class is the number of classes. The value at row i, column j should be the predicted probability of class j for the input number i in the test set. To create the zip file for submission you can use e.g.

zip submission.zip cifar10_probs.csv imdb_probs.csv

To upload your submission, use the participate tab on CodaLab.


Questions?

If you have any questions about the competition, please see our FAQ and feel welcome to contact us at bdlcompetition@gmail.com or send us a DM on Twitter.

<