Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Fallback summary not populated if InMemoryTableScan is present in spark plan #8488

Open
ayushi-agarwal opened this issue Jan 9, 2025 · 0 comments
Labels
bug Something isn't working triage

Comments

@ayushi-agarwal
Copy link
Contributor

ayushi-agarwal commented Jan 9, 2025

Backend

VL (Velox)

Bug description

When plan contains InMemoryTableScan, if in its subplan if there is any node which fallbacks to spark, then that information is not available in Spark UI. @ulysses-you @JkSelf Is this a known issue?

== Fallback Summary ==
(1) InMemoryTableScan: Columnar table cache is disabled

== Physical Plan ==
Execute InsertIntoHadoopFsRelationCommand (40)
+- VeloxColumnarWriteFiles (39)
   :- ^ WriteFilesExecTransformer (34)
   :  +- ^ ProjectExecTransformer (33)
   :     +- ^ InputIteratorTransformer (32)
   :        +- ShuffleQueryStage (30), Statistics(sizeInBytes=134.0 MiB, rowCount=2.42E+5)
   :           +- ColumnarExchange (29)
   :              +- VeloxResizeBatches (28)
   :                 +- ^ ProjectExecTransformer (26)
   :                    +- ^ SortExecTransformer (25)
   :                       +- ^ ProjectExecTransformer (24)
   :                          +- ^ InputIteratorTransformer (23)
   :                             +- RowToVeloxColumnar (21)
   :                                +- TableCacheQueryStage (20), Statistics(sizeInBytes=133.7 MiB, rowCount=2.42E+5)
   :                                   +- InMemoryTableScan (1)
   :                                         +- InMemoryRelation (2)
   :                                               +- AdaptiveSparkPlan (19)
   :                                                  +- Project (18)
   :                                                     +- BroadcastHashJoin LeftOuter BuildRight (17)
   :                                                        :- Project (13)
   :                                                        :  +- Project (12)
   :                                                        :     +- Filter (11)
   :                                                        :        +- Window (10)
   :                                                        :           +- WindowGroupLimit (9)
   :                                                        :              +- Sort (8)
   :                                                        :                 +- Exchange (7)
   :                                                        :                    +- WindowGroupLimit (6)
   :                                                        :                       +- Sort (5)
   :                                                        :                          +- Project (4)
   :                                                        :                             +- Scan hive spark_catalog.raw_www (3)
   :                                                        +- BroadcastExchange (16)
   :                                                           +- Filter (15)
   :                                                              +- Scan hive spark_catalog.stg_www (14)
   +- RowToVeloxColumnar (38)
      +- WriteFiles (37)
         +- NoopLeaf (36)

Spark version

Spark-3.5.x

Spark configurations

No response

System information

No response

Relevant logs

No response

@ayushi-agarwal ayushi-agarwal added bug Something isn't working triage labels Jan 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage
Projects
None yet
Development

No branches or pull requests

1 participant