System Optimization to Accelerate Distributed Model Training
Distributed model training in Shifu/Guagua is proved to be 5-10x faster than Spark MLLib especially in a shared multi-tenancy Hadoop cluster. Several important system-level optimizations are listed here to show why Shifu/Guagua is faster.
