Re-assembling the past: The RePAIR dataset and benchmark for real world 2D and 3D puzzle solving

Theodore Tsesmelis1,*, Luca Palmieri2,*, Marina Khoroshiltseva2, Adeela Islam1, Gur Elkin3, Ofir Itzhak Shahar3, Gianluca Scarpellini1, Stefano Fiorini1, Yaniv Ohayon3, Nadav Alali3, Sinem Aslan2,7, Pietro Morerio1, Sebastiano Vascon2, Elena Gravina4, Maria Cristina Napolitano4, Giuseppe Scarpati4, Gabriel Zuchtriegel4, Alexandra SpĆ¼hler5, Michel E. Fuchs5, Stuart James1,6, Ohad Ben-Shahar3, Marcello Pelillo2, Alessio Del Bue1
*Indicates Equal Contribution
1Fondazione Istituto Italiano di Tecnologia 2Ca' Foscari University of Venice 3Ben-Gurion University of the Negev 4Parco Archeologico di Pompei 5University of Lausanne 6Durham University 7University of Milan
to be presented at   NeurIPS 2024 Track Datasets and Benchmarks

Abstract

This paper proposes the RePAIR dataset that represents a challenging benchmark to test modern computational and data driven methods for puzzle-solving and reassembly tasks. Our dataset has unique properties that are uncommon to current benchmarks for 2D and 3D puzzle solving. The fragments and fractures are realistic, caused by a collapse of a fresco during a World War II bombing at the Pompeii archaeological park. The fragments are also eroded and have missing pieces with irregular shapes and different dimensions, challenging further the reassembly algorithms. The dataset is multi-modal providing hi-res images with characteristic pictorial elements, detailed 3D scans of the fragments and meta-data annotated by the archaeologists. Ground truth has been generated through several years of unceasing fieldwork, including the excavation and cleaning of each fragment, followed by manual puzzle solving by archaeologists of a subset of 1,000 pieces among the 16,000 available. After digitizing all the fragments in 3D, a benchmark was prepared to challenge current reassembly and puzzle-solving methods that often solve more simplistic synthetic scenarios. The tested baselines show that there clearly exists a gap to fill in solving this computationally complex problem.

Resources

Puzzle Solving (Reassembly)

2D Baselines 3D Baselines

A preview of the 3D fragments

A preview of the 2D fragments

Acknowledgment

This work is part of a project that has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No.964854.

eu flag.
logo repair.