OcMotion dataset

[muses0229]

Hi,I wonder will the OcMotion dataset be public?

[tiangexiang]

yes! Sorry for the delay, it is expected to be released before the end of this month.

[tiangexiang]

The preprocessed two OcMotion sequences along with the config files have been released. Please see README for more details! Please let me know if you encounter any problems.

[muses0229]

Thanks for sharing the dataset.Now I can't figure out how to select train dataset and eval dataset for OcMotion.The 0011_02 video has 501 images totally,and since like you choose 501 images for training.But unlike zju_mocap,there is no script to preprocess OcMotion for evaluation,how should I do?

[muses0229]

It seems like OcMotion only have the occluded images,we can't compare the predicted image with the unoccluded image ,so I wonder how to calculate the metrics for the predicted images?

[tiangexiang]

Yes, in real-world data, like OcMotion, there is no GT for the occluded regions. In the paper, we only computed SSIM/PSNR on the visible human parts only (masked by the binary masks). The provided OcMotion data are already preprocessed. Please let me know if you encounter any problems during evaluation.

[muses0229]

Thanks a lot!Now I try to project world 3d pts onto 2d image plane,and the results doesn't look right,seems like the translation is a bit wrong.I am not familiar with the OcMotion dataset,could you please give me some advices?

[muses0229]

[muses0229]

Here is my code to project points,the R,T,K are all cameras' parameters.
RT = torch.cat([torch.tensor(data.R.transpose()), torch.tensor(data.T).reshape(3,1)], -1)[None, None].cuda()
xyz = torch.repeat_interleave(world_3d_pts.squeeze(0)[None, None], repeats=RT.shape[1], dim=1)
xyz = torch.matmul(RT[:, :, None, :, :3].float(), xyz[..., None].float()) + RT[:, :, None, :, 3:].float()
xyz = torch.matmul(torch.tensor(data.K)[None, None][:, :, None].float().cuda(), xyz)[..., 0]
xy = xyz[..., :2] / (xyz[..., 2:] + 1e-5)
src_uv = xy.view(-1, *xy.shape[2:]).squeeze(0) # (6890,2)

[tiangexiang]

Your projection script looks good, did you use the provided official SMPL translation when obtaining the world_3d_pts?

[muses0229]

I use the following code to transform points from smpl space to world space,where R=smpl_params['R'] ,Th=smpl_params['Th']
R_inv = torch.inverse(R)
world_src_pts = torch.matmul(smpl_src_pts, R_inv) + Th
And the extrinsics of camera have transform to world coord using these codes:
E = apply_global_tfm_to_camera(
E=E,
Rh=dst_skel_info['Rh'],
Th=dst_skel_info['Th'])
R = E[:3, :3]
T = E[:3, 3]

[muses0229]

Sorry,I make a mistake.This is without camera global transform:

This one with camera global transform:

[muses0229]

Your projection script looks good, did you use the provided official SMPL translation when obtaining the world_3d_pts?

And if I add smpl_param['trans'] on world_3d_pts,the src_uv will be out of boundary.

[muses0229]

If I remove the global transform of cameras and project without smpl_param['Th'] like this:
src_uv = project_3d_to_2d(data, torch.matmul(smpl_src_pts, R_inv))
I will get:

Seems like still missing a translation,but the provided official SMPL translation didn't work.

[tiangexiang]

Someone has found that using the visualization script from the OcMotion repo may give the correct rendering. Please give it a try. I will look into this issue a bit later myself :)

[muses0229]

sorry for bothering again,but my projection result still seems a bit wrong.I'm pretty sure my projection code is right because I get the same result using ocmotion's code.
Here is my code to transform points from canonical space to observation space,I remove 'Th' and get the above result but still not right.

[hi-zhangjie]

From my previous tests, the SMPL parameters and camera parameters are correct in both the ocmotion dataset and the preprocessed data in this repository.

[muses0229]

Thanks for sharing,I check my code and found I add wrong transpose on camera.R,now my projection result is like that:

Is this looks right now?

[hi-zhangjie]

What is the id of this pic, I will check it later

[imabackstabber]

I'm a little confused by the OcMotion training and testing settings. I found that only frames from a single camera are provided. So it implies that the same picture for training is also used for testing...? Or I must be missing the frames used for testing? Where should I get them?

[muses0229]

Because OcMoton dataset didn't capture unoccluded image,I think testing is to directly rendering the input frames.

[muses0229]

What is the id of this pic, I will check it later

I have fix this,thanks so much!

[imabackstabber]

Because OcMoton dataset didn't capture unoccluded image,I think testing is to directly rendering the input frames.

Ok, thanks a lot! Also, I happened to encounter the same problem, which is a misalignment between SMPL mesh and humans in the 2D images using the parameters provided.(like you described in this issue), would you mind sharing your code snippet? That will help me a lot!

[ZeKai-Jiang]

Thanks for sharing,I check my code and found I add wrong transpose on camera.R,now my projection result is like that: Is this looks right now?

I encounter the same problem, could you explain it more clearly? Thank you very much!