Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PYTHON-4204 Optimize JSON decoding using lookup table to find $ keys #1512

Merged
merged 7 commits into from
Feb 7, 2024

Conversation

ShaneHarvey
Copy link
Member

PYTHON-4204 Optimize JSON decoding using lookup table to find $ keys.

This change improves JSON decoding by 20-50%+:
Screenshot 2024-02-06 at 3 15 29 PM
https://spruce.mongodb.com/task/mongo_python_driver_perf_tests_perf_6.0_standalone_patch_698599c737925ccac5c27cc58bb48b2692c74e30_65c2ad625623431a4f5596c5_24_02_06_22_06_33/trend-charts?execution=0&sortBy=STATUS&sortDir=ASC

I also tested set intersection but the for-loop approach performs better:

def object_hook(dct: Mapping[str, Any], json_options: JSONOptions = DEFAULT_JSON_OPTIONS) -> Any:
    matches = set(dct) & _PARSERS_SET
    if matches:
        return _PARSERS[matches.pop()](dct, json_options)
    return dct

@ShaneHarvey ShaneHarvey requested a review from a team as a code owner February 6, 2024 23:22
@ShaneHarvey ShaneHarvey requested review from NoahStapp and removed request for a team February 6, 2024 23:22
python -m pytest -v --durations=5 --maxfail=10 $TEST_ARGS
# Use --capture=tee-sys so pytest prints test output inline:
# https://docs.pytest.org/en/stable/how-to/capture-stdout-stderr.html
python -m pytest -v --capture=tee-sys --durations=5 --maxfail=10 $TEST_ARGS
Copy link
Member Author

@ShaneHarvey ShaneHarvey Feb 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This lets us see the print()s in the pytest output:

 [2024/02/06 14:45:55.785] + python -m pytest -v --capture=tee-sys --durations=5 --maxfail=10 test/performance/perf_test.py
 [2024/02/06 14:45:56.173] ============================= test session starts ==============================
 [2024/02/06 14:45:56.173] platform linux -- Python 3.10.4, pytest-8.0.0, pluggy-1.4.0 -- /data/mci/6a62d81f9a459de6454c8a15a2244eb2/src/.tox/test-eg/bin/python
 [2024/02/06 14:45:56.173] cachedir: .tox/test-eg/.pytest_cache
 [2024/02/06 14:45:56.173] rootdir: /data/mci/6a62d81f9a459de6454c8a15a2244eb2/src
 [2024/02/06 14:45:56.173] configfile: pyproject.toml
 [2024/02/06 14:45:56.335] collecting ... collected 25 items
 [2024/02/06 14:45:56.442] test/performance/perf_test.py::TestFlatEncoding::runTest Completed TestFlatEncoding 330.739 MB/s, MEDIAN=0.018s, total time=0.183s
 [2024/02/06 14:45:56.666] PASSED          [  4%]
 [2024/02/06 14:45:56.666] test/performance/perf_test.py::TestFlatDecoding::runTest Completed TestFlatDecoding 278.886 MB/s, MEDIAN=0.022s, total time=0.217s
 [2024/02/06 14:45:56.833] PASSED          [  8%]

Copy link
Contributor

@NoahStapp NoahStapp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks Shane!

@ShaneHarvey ShaneHarvey merged commit 4c2e717 into mongodb:master Feb 7, 2024
72 of 76 checks passed
@ShaneHarvey ShaneHarvey deleted the opt-json-decode branch February 7, 2024 00:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants