Ahmad AlMughrabi*, Umair Haroon*, Ricardo Marques* ¹, Petia Radeva* ²
AIBA Lab @ UB (Universitat de Barcelona)*, Computer Vision Center ¹, Institut de Neurosciències ²
Accurate food volume estimation is essential for various applications, including dietary monitoring, medical nutrition management, and food intake analysis. Traditional methods for 3D food volume estimation typically excel at calculating volume but often fall short when it comes to selecting specific food portions.
VolTex addresses this gap by providing an innovative framework that enhances food object selection during volume estimation. Our solution allows users to specify a target food item via text input, enabling the precise segmentation of specific food objects within real-world scenes. Once an object is segmented, our method reconstructs it using Neural Surface Reconstruction techniques, producing high-fidelity 3D meshes for accurate volume calculation.
In our extensive evaluations using the MetaFood3D dataset, VolTex has demonstrated superior performance in isolating and reconstructing food items, leading to improved accuracy in volume estimation. Whether you're a researcher, developer, or enthusiast in the field of food technology, we invite you to explore our framework and contribute to advancing the capabilities of food volume estimation.
- Text-based food item selection for precise segmentation.
- High-fidelity 3D mesh reconstruction using Neural Surface Reconstruction
- Thorough evaluation on the MetaFood3D dataset
We encourage you to check out the documentation for installation instructions, usage details, and contributions. Let’s revolutionize the way we estimate food volume together!
To get started, clone this repository:
git clone https://github.com/GCVCG/VolTex.git
- Python 3.8+
- PyTorch 1.10+
- torchvision
- numpy
- pandas
- scikit-learn
- matplotlib
- OpenCV
You can install the required packages using pip:
pip install -r requirements.txtPlease note that this project relies on several submodules that are not included in this repository. You must clone and install these submodules from their respective repositories:
-
Pixel-Perfect Structure-from-Motion with Featuremetric Refinement
- Repository: Pixel-Perfect-SfM
-
Tracking Anything with Decoupled Video Segmentation (DEVA)
- Repository: DEVA
-
NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction
- Repository: NeuS2
After installing the necessary packages and submodules, you can run the main script for the Pipeline:
bash volETA_NeRF.shWe used the dataset Provided by MetaFood CVPR Workshop Challenge 2024. We used 13 food scenes categorized by difficulty (simple, medium).
You can download the data from Kaggle: MTF Challenge Dataset
A two-phase evaluation process focuses on the precision of reconstructed 3D models in terms of shape and portion size.
- Metric: Mean Absolute Percentage Error (MAPE).
- Focus: Accuracy of volume estimation of 3D models.
- Eligibility: Top teams from Phase-I.
- Requirement: Submission of complete 3D mesh files for each food item.
- Metric: Chamfer distance.
- Focus: Accuracy of 3D shape reconstruction.
This work was partially funded by the EU project MUSAE (No. 01070421), 2021-SGR-01094 (AGAUR), Icrea Academia’2022 (Generalitat de Catalunya), Robo STEAM (2022-1-BG01-KA220-VET000089434, Erasmus+ EU), DeepSense (ACE053/22/000029, ACCIÓ), CERCA Programme/Generalitat de Catalunya, and Grants PID2022141566NB-I00 (IDEATE), PDC2022-133642-I00 (DeepFoodVol), and CNS2022-135480 (A-BMC) funded by MICIU/AEI/10.13039/501100 011033, by FEDER (UE), and by European Union NextGenerationEU/ PRTR. R. Marques acknowledges the support of the Serra Húnter Programme. A. AlMughrabi acknowledges the support of FPI Becas, MICINN, Spain. U. Haroon acknowledges the support of FI-SDUR Becas, MICINN, Spain.