microsoft / microsoft/MoGe

Finetune on Spring dataset questions

Open
#58 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
3k
Forks
234
PR merge metrics
No merged PRs in 30d

Description

Hello, thank you for releasing the training code. :)

I have a few questions. I want to finetune the model on an existing dataset that you noted in your supplementary materials, on the spring dataset. So I did a comparison with the stock version of the model and I obtained some metrics.
Note: The "test" spring dataset was obtained from the hugging face preprocessed datasets you mentioned in the eval section.

After that I took the entire train Spring dataset, and crafted the structure required to finetune the model on the Spring dataset, I followed your advice on the learning rates being 1e-5 for the decoder and 1e-6 for the encoder, and finetuned the model for an exact 1000 iterations ( too low?, it takes a while for a bigger number of iterations )
After doing so, I again reevaluated the new model with the new weights and observed my results are now slightly deviated from the original weights evaluation, being a little worse.

Questions:
Regarding the dataset crafting, in the meta.json structure, I also rescalled the intrinsics from:

{
"intrinsics": [
[
2181.8181818181815,
0,
960.0
],
[
0,
2181.8181818181815,
540.0
],
[
0,
0,
1
]
]
}

To:
{
"intrinsics": [
[
1.1363636363636362,
0,
0.5
],
[
0,
2.02020202020202,
0.5
],
[
0,
0,
1
]
]
}
, having cx and cy to 0.5

Is this normalized version of the intrinsics right?
Are there any tips in finetuning such a large foundation model?
Increasing the number of iterations will also play a huge part here?
Also the weight here, if we have only a dataset plays a part?
"name": "dataset_spring",
"path": "......",
"label_type": "synthetic",
"index": ".index.txt",
"depth": "depth.png",
"weight": 10, <- what role plays this when we have only one dataset? I think it doesn't matter?
"center_augmentation": 0.25,
"fov_range_absolute": [30, 150],
"fov_range_relative": [0.5, 1.0],
"image_augmentation": ["jittering", "jpeg_loss", "blurring", "shot_noise"]

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the supplementary materials and eval section, then inspect the meta.json structure and the Hugging Face preprocessed Spring test dataset used for comparison. Reproduce the stock-versus-finetuned evaluation using the stated 1,000 iterations and recorded intrinsics, and document answers about intrinsic normalization, iteration count, and the single-dataset weight.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface
Domain
computer-vision, machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.