SpaceHack - metric modules

Implementing a new dataset module

  1. Create or claim a GitHub issue from the SpaceHack issue board. that describes the dataset module you want to implement. There are currently around 20 evaluation metrics to implement, but if you come up with a new idea, please create a new issue, add the appropriate tags, and assign the task to yourself.
  2. Add metadata to our metadata spreadsheet on the metrics tab. Please fill in as much as you can as metadata is helpful! If you feel the need, please also add new columns or add additional notes.
  3. Now you are ready to create a new git branch. Try to give your new branch an intuitive prefix such as metric.... You can create a new branch in several ways: (i) create a branch directly from the issue board and then git checkout that branch, or (ii) via the command line:
# clone the template repository
git clone https://github.com/SpatialHackathon/SpaceHack2023.git
# create and switch to a new branch for your e.g. metric "Correx"
git branch metric_correx_naveedishaque # try to make the branch name unique!
git checkout metric_correx_naveedishaque
# link the branch to the issue via the issue board: https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue
  1. Modify the files, filenames, and code in template/, referring to the examples in the metric (for the ARI and V measure metrics).
  2. Test. Before you make a pull request make sure that you (... do something...?). You should try to run your metric on the default method outputs (either SpaGCN or BayesSpace). We are currently working on validators and automatic testing scripts... but this is tricky. Reach out to Niklas Muller-Botticher when you are ready to test!
  3. Create a pull request
  4. Code review (by whom?) and merge your contributed module into the GitHub main branch!

For examples have a look here for ARI or here for the V measure.

Metric module layout and interface

Please define a conda recipe as a yml file for all the dependencies required for your methods script. This should list all the dependencies and also explicitly define the versions.

Input: - Predicted labels - Groundtruth or embedding - Optional parameters (as config file) - …

Output: - Scalar