We used a single image input for simplicity. Our two-stage framework (a) shows the SETR framework, where it accepts an image and generates a mask, followed by (b) XMem2, which accepts the mask and a set of images as a given input and produces masks for all frames.

Checkpoint must be added to the root dir (It must look like FoodMem/ckpts/SETR_MLA/iter_80000.pth): https://drive.google.com/drive/folders/1Bxwj8FDGIdOnEnscjLwB7sisHlMHdo7H?usp=drive_link Saves must be added to the XMem2 dir (It must look like FoodMem/XMem2/saves): https://drive.google.com/drive/folders/1pLiy-hyjzscLjmysexPDmp5DW3QJv0t4?usp=drive_link

conda create -n FoodMem python=3.8

conda activate FoodMem   

conda install pytorch==1.8.1 torchvision==0.9.1 torchaudio==0.8.1 cudatoolkit=11.3 -c pytorch -c conda-forge

pip install -U openmim

mim install mmcv-full==1.7.1

pip install -r requirements.txt

cd XMem2

pip install -r requirements.txt

bash scripts/download_models.sh

bash scripts/download_models_demo.sh

# Build the docker image
docker build -t gcvcg/foodmem .

# Run in Docker
docker run --gpus all -it --rm -e DISPLAY=:1 \ 
-v /tmp/.X11-unix:/tmp/.X11-unix \
-v /path/to/data:/vgdata \
-v $(pwd)/workspace:/app/workspace gcvcg/foodmem bash run.sh /vgdata/$SCENEID

# in case you want to run it in parallel.
# ls -d /vgdata/*/* | parallel -I% --max-args 1  --jobs 1 bash run.sh %

python XMem2/process_video.py --video <path/to/folder> --masks <path/to/folder> --output <path/to/folder>

python .\src\eval_map.py --submit_dir <path/to/folder> --truth_dir <path/to/folder> --output <path/to/folder>

The models include FoodSAM, DEVA, kMean++, and our framework. The inference time was recorded in the format of hours:minutes:seconds.

Comparison of mean average precision scores achieved by different models on two datasets: Nutrition5k and V&F. The models evaluated include FoodSAM, DEVA, kMean++, and our framework.

Comparison of recall scores achieved by different models on two datasets: Nutrition5k and V&F. The models evaluated include FoodSAM, DEVA, kMean++, and our framework.

NOTE: FoodSAM performs better than our framework in the Nutrition5k dataset. This is because FoodSAM was trained on datasets where the camera followed a predefined path to capture images, similar to the setup in the Nutrition5k dataset. On the other hand, our framework performs better in the Vegetables & Fruits dataset, where the camera has freedom of movement, resulting in less predictable image capture scenarios.

A large part of the code is borrowed from the following projects:

1. FoodSAM
2. SETR
3. XMem++

Also mention the following works that helped us to understand and develop our framework:

1. Vision Transformer (ViT)
2. Segment Anything
3. DEVA
4. XMem
5. Nutrition5k
6. Vegetables & Fruits
7. LabelMe
8. Imagededup