ARTIFACT EVALUATION
Reproducibility of experimental results is crucial to foster an atmosphere of trustworthy, open, and reusable research. To improve and reward reproducibility, FormaliSE 2025 continues its Artifact Evaluation (AE) procedure.
An artifact is any additional material (software, data sets, machine-checkable proofs, etc.) that substantiates the claims made in the paper and ideally makes them fully reproducible.
Submission of an artifact is optional but encouraged for all papers where it can support the results presented in the paper. Artifact review is single-anonymous (the paper corresponding to an artifact must still follow the double-anonymous submissions requirements) and will be conducted concurrently with the paper reviewing process. Artifacts will be handled by a separate Artifact Evaluation Committee, and the Artifact Evaluation process will be set up such that the anonymization of the corresponding papers will not be compromised. Accepted papers with a successfully evaluated artefact will be awarded the EAPLS badges that apply (among “Functional”, “Reusable”, and “Available”). Awarded badges are to be added to the camera-ready version of the paper.
Artifacts will be assessed with respect to their consistency with the results presented in the paper, their completeness, their documentation, and their ease of use. The Artifact Evaluation will include an initial check for technical issues; authors of artifacts may be contacted by email within the first two weeks after artifact submission to help resolve any technical problems that prevent the evaluation of an artifact if necessary.
The results of an artifact evaluation will not be available to the reviewers of the corresponding paper; hence, they will not affect the paper’s acceptance decision. However, reviewers will know whether a paper has submitted any artifacts; this piece of information may be taken into account to decide whether the paper should be accepted. Thus, if there are justifiable reasons why a paper’s artifacts cannot be submitted, they should be pointed out in the paper so that the reviewers can appreciate them and adjust their expectations accordingly.
Detailed guidelines for preparation and submission of artifacts will be specified later on.