VolETA: One- and Few-shot Food Volume Estimation

Ahmad AlMughrabi*, Umair Haroon*, Ricardo Marques* ¹, Petia Radeva* ²

AIBA Lab @ UB (Universitat de Barcelona)*, Computer Vision Center ¹, Institut de Neurosciències ²

---

Meta Food CVPR Workshop Challenge 2024: Physically Informed 3D Food Reconstruction

We Won the Meta Food CVPR Workshop Challenge 2024!

We are thrilled to announce that our team has won the prestigious Meta Food CVPR Workshop Challenge 2024! This achievement is a testament to our hard work, innovative methodologies, and dedication to advancing the field of food recognition using computer vision.

---

Table of Contents

• Introduction
• Installation
• Usage
• Methodology
• Submodules
• Results
• License
• Acknowledgements

---

Introduction

Accurate dietary assessment using 2D images is crucial but challenging due to the need for precise food volume estimation. Our approach addresses these challenges by providing a semi-automated method that is adaptable and user-friendly.

Challenges in Food Volume Estimation

• 2D Image Limitations: Lack of depth information in standard 2D images.
• Reference Object Methods:
  • Use standard-sized items or specific markers for 3D cues.
  • Computationally intensive and often impractical.
• Non-Reference Object Methods:
  • Estimate depth using image features and camera properties.
  • Variable camera orientations and positions pose limitations.

Our Contributions

• Unbounded Food Scenes:
  • Flexible camera movement around the food object.
  • Handles varying capturing speeds and topologies.
• Sparse Input Views:
  • Requires only one or a few RGBD images.

Our Framework

Installation:

To get started, clone this repository:

git clone https://github.com/umairharon/VolETA-MetaFood

Requirements

• Python 3.8+
• PyTorch 1.10+
• torchvision
• numpy
• pandas
• scikit-learn
• matplotlib
• OpenCV

You can install the required packages using pip:

pip install -r requirements.txt

Submodules

Please note that this project relies on several submodules that are not included in this repository. You must clone and install these submodules from their respective repositories:

1. Pixel-Perfect Structure-from-Motion with Featuremetric Refinement

   • Repository: Pixel-Perfect-SfM

2. Segment Anything (SAM)

   • Repository: Segment-Anything

3. XMem++: Production-level Video Segmentation From Few Annotated Frames

   • Repository: XMem2

4. NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction

   • Repository: NeuS2

Usage

After installing the necessary packages and submodules, you can run the main script to the Pipeline:

python main.py --config configs/config.yaml #Example

Methodology

Data:

We used the dataset Provided by MetaFood CVPR Workshop Challenge 2024. It comprises 20 food scenes categorized by difficulty (simple, medium, hard).

You can download the data from Kaggle: MTF Challenge Dataset

Evaluation

A two-phase evaluation process focuses on the precision of reconstructed 3D models in terms of shape and portion size.

Phase-I (Portion Size Evaluation):

• Metric: Mean Absolute Percentage Error (MAPE).
• Focus: Accuracy of volume estimation of 3D models.

Phase-II (Shape Evaluation):

• Eligibility: Top teams from Phase-I.
• Requirement: Submission of complete 3D mesh files for each food item.
• Metric: Chamfer distance.
• Focus: Accuracy of 3D shape reconstruction.

Results

Phase-I

• MAPE: 0.10973

Phase-II

• Chamfer Distance With Transformation Matrix:
  • Average: 0.007258650766
  • Sum: 0.13066
• Chamfer Distance Without Transformation Matrix:
  • Average: 0.09528961389
  • Sum: 1.71521

Visual Results

Comparison Over Ground Truth

Acknowlegements

This work was partially funded by the EU project MUSAE (No. 01070421), 2021-SGR-01094 (AGAUR), Icrea Academia’2022 (Generalitat de Catalunya), Robo STEAM (2022-1-BG01-KA220-VET000089434, Erasmus+ EU), DeepSense (ACE053/22/000029, ACCIÓ), CERCA Programme/Generalitat de Catalunya, and Grants PID2022141566NB-I00 (IDEATE), PDC2022-133642-I00 (DeepFoodVol), and CNS2022-135480 (A-BMC) funded by MICIU/AEI/10.13039/501100 011033, by FEDER (UE), and by European Union NextGenerationEU/ PRTR. R. Marques acknowledges the support of the Serra Húnter Programme. A. AlMughrabi acknowledges the support of FPI Becas, MICINN, Spain. U. Haroon acknowledges the support of FI-SDUR Becas, MICINN, Spain.