kiez-benchmarking

Configurations and results of kiez paper

Install dependencies

You can find the necessary depencies in the pyproject.toml. If you have poetry installed simply execute:

poetry install

Pre-calculated knowledge graph embeddings

The knowledge graph embeddings produced for and used in our study are available via Zenodo.

Pre-calculated results

We make our results available in results/max_all.csv. Our plots can be created with:

poetry run python kiezbenchmarking/create_plots.py --use-csv output

We make a lot more plots available in results/additional_report.pdf. To recreate this document:

poetry run python kiezbenchmarking/create_plots.py --extensive plot_dir
poetry run python kiezbenchmarking/create_report.py plot_dir results/
cd results
pdflatex additional_report.tex

Reproduce results

It is easy to run a single experiment:

poetry run python kiezbenchmarking/experiment.py --embedding "AttrE" --dataset "D_W_15K_V1" --neighbors 50 faiss --candidates 100 --index-key Flat --no-gpu ls --method nicdm

This will automatically download any data if necessary.

You can also track your results with wandb using the --use-wandb flag, e.g.:

poetry run python kiezbenchmarking/experiment.py --embedding "AttrE" --dataset "D_W_15K_V1" --neighbors 50 --use-wandb faiss --candidates 100 --index-key Flat --no-gpu ls --method nicdm

This command shows you the necessary arguments to run an experiment:

poetry run python kiezbenchmarking/experiment.py --help

The individual nearest neighbor algorithm and hubness reduction method are declared via subcommand, for which you can also get help (after supplying the required arguments of the base command):

poetry run python kiezbenchmarking/experiment.py --embedding "AttrE" --dataset "D_W_15K_V1" --neighbors 50 faiss --help

or for the hubness reduction method:

poetry run python kiezbenchmarking/experiment.py --embedding "AttrE" --dataset "D_W_15K_V1" --neighbors 50 faiss --candidates 100 --index-key Flat --use-gpu False ls --help

For archival purposes: Using SEML

We originally used SEML to keep track of results. Please refer to their instructions to set everything up. Install the necessary packages inside a conda env (because seml wants you to use a conda env):

conda create -n kiez python=3.7.1
conda activate
poetry install
conda deactivate

Queue the experiments

seml [db_name] add configs/[path_to_config]

Run them

seml [db_name] run

Which starts a SLURM job with all the experiments and saves the results in your MongoDB using Sacred.