Deepfake Benchmark Results: What They Can and Cannot Tell You

Start with the question the benchmark was built to answer

A benchmark result can look like a ready-made verdict: one system scored higher, so it must be better at deciding whether any image or video is fake. That conclusion reaches beyond the evidence. A benchmark measures a system within a particular evaluation. Before using the result, identify what the evaluation asked the system to do.

NIST describes the Open Media Forensics Challenge Evaluation, or OpenMFC, as an open evaluation series for assessing and measuring media-forensics algorithms and systems. Its stated focus is automated manipulation detection and localization for images and video. The tasks include deciding whether media is manipulated and, when applicable, identifying the region and type of manipulation. The program also supports tasks involving GAN manipulation detection.

Those statements define the subject of the evaluation. They do not supply a universal accuracy claim for every detector, every manipulation, or every file a reader may encounter. The useful reading question is narrower: what capability did this evaluation measure, and under which published task?

Read the task before the result

Begin with the task name and its output. Detection, localization, and manipulation-type identification answer different questions. A result for one should not silently become evidence for another. If a table or leaderboard is separated by task, keep that separation in your notes.

Next, find the material that defines the evaluation. NIST says OpenMFC provides benchmark datasets and evaluation infrastructure, and directs prospective participants to its program website. The public page also says that NIST-released datasets are available upon request after signup and completion of license agreements. If the result you are reading points to additional documentation, use that documentation to identify the relevant data, task, and reporting terms. If those details are unavailable, record the gap instead of filling it with assumptions.

Then check whether the result belongs to OpenMFC or to an earlier program. NIST says OpenMFC builds on experience from the Media Forensic Challenge evaluation developed for DARPA's MediFor program. Similar names do not make separate evaluations interchangeable. Preserve the program name attached to the result.

This reading process is an operational recommendation from this article. NIST defines its program and tasks; it does not provide the decision workflow below or claim that a benchmark result settles the authenticity of an individual file.

Keep benchmark evidence separate from a file decision

A benchmark result describes performance reported within that evaluation. An individual file presents a different question: what evidence is available for this image or video, and what action is justified now? Keep the benchmark result as background evidence about a system. Do not copy it into the case record as the file's verdict.

When reviewing a suspicious file, preserve the original if available and record where it came from, who supplied it, and what claim accompanies it. Check whether a trustworthy publisher or original account provides the same material and context. Look for available provenance information. These steps do not depend on guessing which benchmark row best matches the file.

If a detector is used, keep its output with the file and the other evidence. Automated detection can produce false positives and false negatives: authentic material may be flagged, and manipulated material may be missed. A high risk signal supports further review; a low signal does not authenticate the file.

Build a review record another person can repeat

For each benchmark claim you plan to cite, record the program, task, source URL, and the exact result being discussed. Add the date you accessed the material. If the result depends on a table or document that you cannot inspect, say so. A short, bounded note is more useful than a confident summary whose scope cannot be checked.

For an individual media review, keep a separate record: the original file or best available copy, its source and surrounding claim, any provenance evidence, the detector output, and the final human decision. Separating these records prevents a system-level result from quietly becoming proof about one file.

DeepFakeCheck can analyze a saved image, video, audio, or text file and return a probabilistic risk signal. It does not reproduce an OpenMFC evaluation, identify the person who supplied a file, or establish that the depicted event happened. Use its output as one entry in the file review, not as a substitute for source verification.

Use the result without handing it the final decision

A benchmark is useful when the comparison stays inside its stated scope. It can help a reader understand which task was evaluated and locate the program materials behind a reported result. It becomes misleading when a rank or score is treated as a guarantee about unrelated files, tasks, or operating conditions.

Before acting, ask whether the evidence in front of you concerns the evaluated system or the individual file. For the system, cite the exact program and task. For the file, document its source, context, provenance, and any detector output. If the available evidence does not answer the decision you face, keep the conclusion open and continue verification.

The practical boundary is simple to apply: benchmark evidence can inform tool selection and review planning, while the decision about a particular file must remain tied to that file's own evidence.

Sources

• NIST, Open Media Forensics Challenge Evaluation: https://www.nist.gov/itl/iad/mltg/open-media-forensics-challenge