Code

Add unittest for ray text dedup#540

Code

Open

chenyushuo wants to merge 15 commits intomodelscope:mainmodelscope/data-juicer:mainfrom chenyushuo:ray_dedup_with_actorchenyushuo/data-juicer:ray_dedup_with_actorCopy head branch name to clipboard

+1,972-1

Commits

Commits on Jan 2, 2025

• add actors for ray dedup

  chenyushuo

  committed
  eadf93b

  Copy full SHA for eadf93b

Commits on Jan 3, 2025

• code format

  chenyushuo

  committed
  051493b

  Copy full SHA for 051493b

• doc format

  chenyushuo

  committed
  4fc12a6

  Copy full SHA for 4fc12a6

Commits on Jan 7, 2025

• fix in ray dedup

  chenyushuo

  committed
  a6fcd6e

  Copy full SHA for a6fcd6e

Commits on Jan 9, 2025

• Merge branch 'main' of github.com:modelscope/data-juicer into ray_dedup_with_actor

  chenyushuo

  committed
  891219f

  Copy full SHA for 891219f

Commits on Jan 10, 2025

• Merge branch 'main' of github.com:modelscope/data-juicer into ray_dedup_with_actor

  chenyushuo

  committed
  f475338

  Copy full SHA for f475338

• add unittest for ray text dedup

  chenyushuo

  committed
  06b1730

  Copy full SHA for 06b1730

Commits on Jan 13, 2025

• add TEST_TAG ray and operator level test

  chenyushuo

  committed
  35cf0e7

  Copy full SHA for 35cf0e7

• add test_ray_minhash_dedup, test_ray_compute_stats_single_filter and test_ray_compute_stats_batched_filter

  chenyushuo

  committed
  a0f72fb

  Copy full SHA for a0f72fb

Commits on Jan 17, 2025

• add TEST_TAG for RayBTSDedup

  chenyushuo

  committed
  218742c

  Copy full SHA for 218742c

• Merge branch 'main' of github.com:modelscope/data-juicer into ray_dedup_with_actor

  chenyushuo

  committed
  9e94c19

  Copy full SHA for 9e94c19

• doc fix

  chenyushuo

  committed
  ecc4c0a

  Copy full SHA for ecc4c0a

Commits on Jan 20, 2025

• fix in bts test

  chenyushuo

  committed
  24a8640

  Copy full SHA for 24a8640

• Merge branch 'ray_dedup_with_actor' of github.com:chenyushuo/data-juicer into ray_dedup_with_actor

  chenyushuo

  committed
  faec9f2

  Copy full SHA for faec9f2

• merge

  chenyushuo

  committed
  46ba4dd

  Copy full SHA for 46ba4dd