Support GCS #73
Support GCS #73
Conversation
dpxcc
force-pushed
the
main
branch
2 times, most recently
from
2af4b2b to
6a655ea
Compare
raghunandanbhat
force-pushed
the
gcs-support
branch
2 times, most recently
from
8ffabb2 to
03ea259
Compare
1. Create delta tables in GCS 2. Insert/Update/Delete support
raghunandanbhat
force-pushed
the
gcs-support
branch
from
03ea259 to
6892f98
Compare
|
@zhousun please review when you get a chance. thanks! |
|
@raghunandanbhat thanks for the contribute. For delta-rs, I think it is possible to inline the secret as json instead of providing a path: https://docs.rs/object_store/latest/src/object_store/gcp/builder.rs.html#365 |
|
delta-rs now uses the service account json instead of the file path. I believe it is better to provide the path when creating/inserting a secret into |
|
If I understand correct, providing a path like this requires the files to be in server side. For example if you connect to a remote postgres, you cannot use a secret file in local path. That's why I believe having a way to direct type/copy the secret is important. |
README.md
Outdated
| #### GCS Buckets | ||
| - Add your GCS credentials | ||
| ```sql | ||
| SELECT mooncake.create_secret('<name>', 'GCS', '<HMAC_key_id>', '<HMAC_secret>', '{"PATH":"path/to/service_account.json"}'); | ||
| ``` | ||
| - Set default bucket after adding GCS credentials. | ||
| ```sql | ||
| SET mooncake.default_bucket = 'gs://<gcs_bucket>'; | ||
| ``` | ||
| >**Note**: delta-rs seems to accept only Service Account JSON credentials when creating delta tables in GCS. | ||
|
|
There was a problem hiding this comment.
nit: To keep readme more condense, we will move the bucket instruction to https://pgmooncake.com/docs/cloud-storage. :)
There was a problem hiding this comment.
sure.
please make sure to update the sql query.
SELECT mooncake.create_secret('<name>', 'GCS', '<HMAC_key_id>', '<HMAC_secret>', '{"PATH":"path/to/service_account.json"}');to
SELECT mooncake.create_secret('<name>', 'GCS', '<HMAC_key_id>', '<HMAC_secret>', '{"GCS_SECRET":"Service Account JSON"}');
what if we keep both options?
|
Yep that would work |
raghunandanbhat
force-pushed
the
gcs-support
branch
from
db60953 to
b4b9472
Compare
rust_extensions/delta/src/lib.rs
Outdated
| @@ -48,8 +55,10 @@ pub fn DeltaCreateTable( | |||
| runtime.block_on(async { | |||
| let mut storage_options: HashMap<String, String> = | |||
| serde_json::from_str(options.to_str()?).expect("invalid options"); | |||
| // Write directly to S3 without locking is safe since Mooncake is the only writer | |||
| storage_options.insert("AWS_S3_ALLOW_UNSAFE_RENAME".to_string(), "true".to_string()); | |||
| if get_storage_type(path) == StorageType::S3 { | |||
There was a problem hiding this comment.
Thinking about this again, I think it might be better to simply return the type from ColumnstoreMetadata::SecretsSearchDeltaOptions() and pass it down here
There was a problem hiding this comment.
yes, it is much simpler. we can return tuple<string, string> from ColumnstoreMetadata::SecretsSearchDeltaOptions() and pass them separately to DeltaCreateTable()
1. Support raw JSON secrets and path based secret for GCS 2. Bugfix to pick correct secret based on longest matching scope
raghunandanbhat
force-pushed
the
gcs-support
branch
from
b4b9472 to
62a1f99
Compare
fixes #38