Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Extract plan features #55094

Open
wants to merge 17 commits into
base: main
Choose a base branch
from
Open

[Feature] Extract plan features #55094

wants to merge 17 commits into from
+783 −17

Conversation

murphyatwork
Copy link
Contributor

@murphyatwork murphyatwork commented Jan 15, 2025 •
edited
Loading

Why I'm doing:

What I'm doing:

Extract plan features into fe.features.log.

Fixes #55096

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto-backported to the target branch
    • 3.4
    • 3.3
    • 3.2
    • 3.1
    • 3.0

@murphyatwork
refine cost model for certain operators
b1ab8dc 
Signed-off-by: Murphy <[email protected]>
@murphyatwork
simplify the cost
a095dcd 
Signed-off-by: Murphy <[email protected]>
@murphyatwork
record plan features into log
b0ac3f9 
Signed-off-by: Murphy <[email protected]>
@murphyatwork
tmp
2ddbc84 
Signed-off-by: Murphy <[email protected]>
@murphyatwork
refactor
1a88f7f 
Signed-off-by: Murphy <[email protected]>
@murphyatwork
add join operator features
cf39c20 
Signed-off-by: Murphy <[email protected]>
@murphyatwork
add agg operator features
83f0c9c 
Signed-off-by: Murphy <[email protected]>
@murphyatwork
add env features
7658159 
Signed-off-by: Murphy <[email protected]>
@murphyatwork
record features only for SELECT
01d7566 
Signed-off-by: Murphy <[email protected]>
@murphyatwork
change config
e831aee 
Signed-off-by: Murphy <[email protected]>
@murphyatwork
add exchange operator features
2c4fa8a 
Signed-off-by: Murphy <[email protected]>
@murphyatwork
add avg row size as feature
20246eb 
Signed-off-by: Murphy <[email protected]>
@murphyatwork
apply transformation encoding
d712d68 
Signed-off-by: Murphy <[email protected]>
@murphyatwork
change log format
afc469f 
Signed-off-by: Murphy <[email protected]>
@murphyatwork murphyatwork requested a review from a team as a code owner January 15, 2025 05:05
@murphyatwork
change log4j config
184154e 
Signed-off-by: Murphy <[email protected]>
@github-actions GitHub Actions
Copy link

[FE Incremental Coverage Report]

✅ pass : 29 / 34 (85.29%)

file detail

path covered_line new_line coverage not_covered_line_detail
🔵 com/starrocks/common/AuditLog.java 0 2 00.00% [62, 63]
🔵 com/starrocks/sql/optimizer/operator/physical/PhysicalOlapScanOperator.java 1 4 25.00% [144, 145, 146]
🔵 com/starrocks/sql/optimizer/Optimizer.java 7 7 100.00% []
🔵 com/starrocks/plugin/AuditEvent.java 2 2 100.00% []
🔵 com/starrocks/common/Log4jConfig.java 9 9 100.00% []
🔵 com/starrocks/sql/Explain.java 2 2 100.00% []
🔵 com/starrocks/sql/optimizer/operator/OperatorType.java 3 3 100.00% []
🔵 com/starrocks/sql/optimizer/cost/CostModel.java 5 5 100.00% []

@murphyatwork
fix code style
17346a7 
Signed-off-by: Murphy <[email protected]>
satanson
satanson previously approved these changes Jan 15, 2025
View reviewed changes
kangkaisen
kangkaisen reviewed Jan 15, 2025
View reviewed changes
fe/fe-core/src/main/java/com/starrocks/sql/optimizer/cost/CostModel.java
// ScanOperator use a ChunkBuffer to implement async-io, which may takes significant memory for certain
// cases
SessionVariable variables = ConnectContext.get().getSessionVariable();
double memCost = (double) statistics.getAvgRowSize() * variables.getDegreeOfParallelism();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to multiply by the bytes of each row of data?

@sonarqubecloud SonarQubeCloud
Copy link

Quality Gate Passed Quality Gate passed

Issues
28 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@github-actions GitHub Actions
Copy link

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

@github-actions GitHub Actions
Copy link

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kangkaisen kangkaisen kangkaisen left review comments

@satanson satanson satanson left review comments

At least 2 approving reviews are required to merge this pull request.

Assignees

@murphyatwork murphyatwork

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Plan Feature Extraction
3 participants
@murphyatwork @satanson @kangkaisen