XGBoost distributed with spark uses only one forrest? Why don't use parallel forrests?

[Anisalexvl]

I found out that using Spark XGBoost results in only a single learned forest, and I don't understand why. It seems we could train different boosted models on each partition and then average their predictions.

This seems misleading to me, because there doesn’t appear to be any real advantage to splitting a large dataset. The result would essentially be the same as if I just subsampled the entire dataset and didn’t use the Spark implementation at all.

xgboost/python-package/xgboost/spark/core.py

Lines 1161 to 1168 in 4500941

	if context.partitionId() == 0:
	config = booster.save_config()
	yield pd.DataFrame({"data": [config]})
	booster_json = booster.save_raw("json").decode("utf-8")
	for offset in range(0, len(booster_json), _MODEL_CHUNK_SIZE):
	booster_chunk = booster_json[offset : offset + _MODEL_CHUNK_SIZE]
	yield pd.DataFrame({"data": [booster_chunk]})

[trivialfis]

XGB is based on a collective (MPI) framework, and workers communicate with each other during training. In the end, each worker returns the same model based on data from all workers. The spark interface uses the result from the first worker as a convention.

[Anisalexvl]

Cannot understand which code is reliable for communication between spark nodes during training. Could you please point to this functionality?

[trivialfis]

The most important communication is allreduce with the gradient histogram. There are other synchronizations as well. Feel free to grep the codebase for allreduce calls.

[trivialfis]

Closing as the original question is solved.