bytedance / bytedance/WMP

how to use "video_pred()" function to predict image

Open
#9 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
381
Forks
35
PR merge metrics
No merged PRs in 30d

Description

def video_pred(self, data):
data = self.preprocess(data)
embed = self.encoder(data)

states, _ = self.dynamics.observe(
embed[:6, :5], data["action"][:6, :5], data["is_first"][:6, :5]
)
recon = self.heads["decoder"](self.dynamics.get_feat(states))["image"].mode()[:6]

reward_post = self.heads["reward"](self.dynamics.get_feat(states)).mode()[:6]
init = {k: v[:, -1] for k, v in states.items()}
prior = self.dynamics.imagine_with_action(data["action"][:6, 5:], init)
openl = self.heads["decoder"](self.dynamics.get_feat(prior))["image"].mode()
reward_prior = self.heads["reward"](self.dynamics.get_feat(prior)).mode()
# observed image is given until 5 steps
model = torch.cat([recon[:, :5], openl], 1)
truth = data["image"][:6]
model = model
error = (model - truth + 1.0) / 2.0

return torch.cat([truth, model], 2)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start from the shown video_pred function and inspect the expected structure of its data argument, including image, action, and is_first. Clarify the intended inputs, outputs, and prediction workflow, then document a reproducible usage example and expected result.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.