Apple M1 slow using GPU

[gerlis22]

I installed the new 2.11 version of Keras and ran predictions on a model I created using Keras 2.9. Predictions with version 2.11 are considerably slower than when I used version 2.9. I'm using MacOS with apple silicone and have GPU working. How can go back to 2.9 version? Any ideas about what could be happening?

R version 4.2.2 (2022-10-31)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Monterey 12.6

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] magrittr_2.0.3 lubridate_1.9.0 timechange_0.2.0 caret_6.0-93 lattice_0.20-45 forcats_0.5.2 stringr_1.5.0
[8] dplyr_1.0.10 purrr_1.0.1 readr_2.1.3 tidyr_1.2.1 tibble_3.1.8 ggplot2_3.4.0 tidyverse_1.3.2
[15] reticulate_1.27 data.table_1.14.6 tensorflow_2.11.0 keras_2.11.0

loaded via a namespace (and not attached):
[1] nlme_3.1-161 fs_1.5.2 httr_1.4.4 rprojroot_2.0.3 tools_4.2.2 backports_1.4.1
[7] utf8_1.2.2 R6_2.5.1 rpart_4.1.19 DBI_1.1.3 colorspace_2.0-3 nnet_7.3-18
[13] withr_2.5.0 tidyselect_1.2.0 compiler_4.2.2 cli_3.6.0 rvest_1.0.3 xml2_1.3.3
[19] scales_1.2.1 tfruns_1.5.1 digest_0.6.31 base64enc_0.1-3 pkgconfig_2.0.3 parallelly_1.34.0
[25] dbplyr_2.3.0 rlang_1.0.6 readxl_1.4.1 rstudioapi_0.14 generics_0.1.3 jsonlite_1.8.4
[31] ModelMetrics_1.2.2.2 googlesheets4_1.0.1 Matrix_1.5-3 Rcpp_1.0.9 munsell_0.5.0 fansi_1.0.3
[37] lifecycle_1.0.3 stringi_1.7.12 whisker_0.4.1 pROC_1.18.0 MASS_7.3-58.1 plyr_1.8.8
[43] recipes_1.0.4 grid_4.2.2 parallel_4.2.2 listenv_0.9.0 crayon_1.5.2 haven_2.5.1
[49] splines_4.2.2 hms_1.1.2 zeallot_0.1.0 pillar_1.8.1 future.apply_1.10.0 reshape2_1.4.4
[55] codetools_0.2-18 stats4_4.2.2 reprex_2.0.2 glue_1.6.2 modelr_0.1.10 png_0.1-8
[61] vctrs_0.5.1 tzdb_0.3.0 foreach_1.5.2 cellranger_1.1.0 gtable_0.3.1 future_1.30.0
[67] assertthat_0.2.1 gower_1.0.1 prodlim_2019.11.13 broom_1.0.2 class_7.3-20 survival_3.5-0
[73] googledrive_2.0.0 gargle_1.2.1 timeDate_4022.108 iterators_1.0.14 hardhat_1.2.0 lava_1.7.1
[79] globals_0.16.2 ellipsis_0.3.2 here_1.0.1 ipred_0.9-13

[t-kalinowski]

I'm not sure, this is the first report here of a performance regression in 2.11.
How was the model saved? How are you loading the model? If you could put together a short reprex it would be helpful for diagnosing.

If the issue is caused by something specific to the tensorflow-macos package, then you might be able to find better help at https://developer.apple.com/forums/tags/tensorflow-metal. This repo is more for issues specific to the R interface.

[gerlis22]

@t-kalinowski thank you for the quick response. I don't know what I did, but speeds are back to normal using keras 2.11. The new issue I have is a bug using tensorflow-macos 2.11 and tensorflow-metal 0.7, where the new optimizer API is forcing an XLA path that has not been implemented yet (https://developer.apple.com/forums/thread/721735).

For example, when I run this tutorial https://tensorflow.rstudio.com/examples/cifar10_cnn
and start training the model, I get the error below. The current workaround is to use tensorflow-macos 2.10 and tensorflow-metal 0.6. How do I install a specific version of ternsorflow-macos and tensorflow-metal in R? It is unclear to me how to do it with install_keras() or install_tensorflow() functions.

2023-01-19 13:39:53.761946: W tensorflow/tsl/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
Epoch 1/50
2023-01-19 13:39:54.107050: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:114] Plugin optimizer for device_type GPU is enabled.
2023-01-19 13:39:54.346180: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:418 : NOT_FOUND: could not find registered platform with id: 0x298f09bc0
2023-01-19 13:39:54.346225: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:418 : NOT_FOUND: could not find registered platform with id: 0x298f09bc0
2023-01-19 13:39:54.357375: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:418 : NOT_FOUND: could not find registered platform with id: 0x298f09bc0
2023-01-19 13:39:54.357408: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:418 : NOT_FOUND: could not find registered platform with id: 0x298f09bc0
2023-01-19 13:39:54.374472: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:418 : NOT_FOUND: could not find registered platform with id: 0x298f09bc0
2023-01-19 13:39:54.374497: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:418 : NOT_FOUND: could not find registered platform with id: 0x298f09bc0
2023-01-19 13:39:54.381408: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:418 : NOT_FOUND: could not find registered platform with id: 0x298f09bc0
2023-01-19 13:39:54.381438: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:418 : NOT_FOUND: could not find registered platform with id: 0x298f09bc0
2023-01-19 13:39:54.401779: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:418 : NOT_FOUND: could not find registered platform with id: 0x298f09bc0
2023-01-19 13:39:54.401811: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:418 : NOT_FOUND: could not find registered platform with id: 0x298f09bc0
2023-01-19 13:39:54.407895: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:418 : NOT_FOUND: could not find registered platform with id: 0x298f09bc0
2023-01-19 13:39:54.407945: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:418 : NOT_FOUND: could not find registered platform with id: 0x298f09bc0
Error in py_call_impl(callable, dots$args, dots$keywords) : 
  tensorflow.python.framework.errors_impl.NotFoundError: Graph execution error:
<... omitted ...>ges/keras/optimizers/optimizer_experimental/optimizer.py", line 1166, in _internal_apply_gradients
      return tf.__internal__.distribute.interim.maybe_merge_call(
    File "/Users/gerardocelis/Library/r-miniconda-arm64/envs/r-reticulate/lib/python3.8/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1216, in _distributed_apply_gradients_fn
      distribution.extended.update(
    File "/Users/gerardocelis/Library/r-miniconda-arm64/envs/r-reticulate/lib/python3.8/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1211, in apply_grad_to_update_var
      return self._update_step_xla(grad, var, id(self._var_key(var)))
Node: 'StatefulPartitionedCall_10'
could not find registered platform with id: 0x298f09bc0
	 [[{{node StatefulPartitionedCall_10}}]] [Op:__inference_train_function_1605]
See `reticulate::py_last_error()` for details

[t-kalinowski]

To install a specific version of tensorflow-macos, your best bet is to use reticulate::py_install(), e.g.:

reticulate::py_install(
  c("tensorflow-macos==2.10.*", "tensorflow-metal==0.7.*"),
    envname = "r-tensorflow",
    pip = TRUE, 
    ignore_installed = TRUE)