Evaluation of the submissions will be performed on a private test data set containing individual files of surgical videos @ 1FPS through grand-challenge automated docker submission and evaluation system. The test data and labels are hidden from the participants. Following are the metrics that will be used to evaluate algorithm performances and displayed on the leaderboards. 

Category 1: Surgical tool detection

The standard COCO dataset bounding box detection evaluation metric (mAP@[0.5:0.05:0.95] - https://cocodataset.org/#detection-eval) will be used for evaluations and determining the winners

Category 2: Surgical visual question answering

For each question, five different answers will be provided as ground truth. For example, the question "Was a large needle driver used during the surgery?"

may have the following list as ground truth answers:

[
"No",
"No, a large needle driver was not used.",
"A large needle driver was not used.",
"No large needle driver was utilized.",
"There is no indication a large needle driver was used."
]

You can download 10 video samples with questions and answers in the same format used for evaluation to be used as reference here

Your algorithm should generate a single predicted answer for each question. The evaluation container will then compute the BERTScore-F1 between your predicted answer and each of the five ground-truth answers. The highest BERTScore-F1 among these five comparisons will be used as the final score for that question.

The platform will get the mean of all calculated max BERTScore-F1 values for each Q&A pair. The evaluation container outputs a metrics.json file with the following structure:

{
    "results": [
        {
            "case_id": "case001",
            "bertscore_f1": 0.723456789
        },
        {
            "case_id": "case002",
            "bertscore_f1": 0.812345678
        },
        {
            "case_id": "case003",
            "bertscore_f1": 0.654321098
        }
    ],
    "aggregates": {
        "bertscore_f1": 0.730041188
    }
}

The BERTScore-F1 is a metric commonly used to evaluate the quality of text generated by models, such as in machine translation and text generation tasks. In our evaluation, it measures how closely your model's predicted answer matches a set of reference answers by comparing contextual embeddings from a pre-trained BERT model.

We use BERTScore with the roberta-large model, which computes precision, recall, and F1 scores based on cosine similarity between contextual embeddings of tokens. The F1 score is used as the primary metric, which balances precision and recall. This approach captures semantic similarity better than traditional n-gram overlap metrics like BLEU, as it understands the contextual meaning of words and phrases.

The scores are rescaled with a baseline to provide interpretable values across different text lengths and domains.

References:

Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019). BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations (ICLR). Retrieved from https://arxiv.org/abs/1904.09675

Sample BERT evaluation code snippet:

import pandas as pd
from bert_score import score

# Load the CSV file
# Assume the CSV has columns 'Question' and 'Answer'
df = pd.read_csv('answers.csv')

# Group answers by question
grouped = df.groupby('Question')['Answer'].apply(list).reset_index()

def calculate_scores(group):
    question = group['Question']
    answers = group['Answer']

    # Use first answer as candidate, rest as references
    candidate = [answers[0]]
    references = [answers[1:]] if len(answers) > 1 else [answers[0:1]]

    # BERT Score
    P, R, F1 = score(candidate, references[0], lang='en', verbose=True, model_type='roberta-large')
    bertscore_f1 = F1.mean().item()

    return pd.Series({'BERT Score': bertscore_f1})

# Apply the scoring function to each group
grouped['BERT Score'] = grouped.apply(calculate_scores, axis=1)

# Display the results
print(grouped[['Question', 'BERT Score']])

Evaluation containers for transparency purposes

Category 1 (same as 2024 and 2025 challenges): https://github.com/isi-challenges/surgvu24-category-1-eval

Category 2: https://github.com/isi-challenges/surgvu26-category-2-eval-public