ECR returns HTTP/400 errors when using aggregators #1938
Comments
captainfalcon23
changed the title
|
Hi @captainfalcon23, My apologies for the delayed response. I have executed the query as outlined in the issue description, ensuring the correct credential configuration, Steampipe CLI version, and plugin version. However, I encountered a syntax error initially. After rectifying the syntax error, I attempted the query multiple times using an aggregator connection, and in all instances, I received results. Credential configuration file( Query: with latest_image_ts as (
select
repository_name,
max(image_pushed_at) as image_pushed_at
from
aws_all.aws_ecr_image
group by
repository_name
),
images_with_tags as (
select
_ctx,
region,
repository_name,
image_pushed_at,
jsonb_array_elements_text(image_tags)::text as image_tag
from
aws_all.aws_ecr_image
)
select
i._ctx ->> 'connection_name' as "AWS_ACCOUNT_NAME",
i.region,
i.repository_name,
i.image_pushed_at,
i.image_tag,
f.name,
f.uri,
f.severity,
f.description,
(
jsonb_path_query(f.attributes, '$[*] ? (@.Key == "package_name")') -> 'Value' #>> '{}'
)::text as package_name,
(
jsonb_path_query(
f.attributes,
'$[*] ? (@.Key == "package_version")'
) -> 'Value' #>> '{}'
)::text as package_version
from
images_with_tags i
join latest_image_ts l on (l.repository_name, l.image_pushed_at) = (i.repository_name, i.image_pushed_at)
join aws_all.aws_ecr_image_scan_finding f on (f.repository_name, f.image_tag) = (i.repository_name, i.image_tag)
order by
i.repository_name,
i.image_tag,
f.severity,
f.name,
package_name;Result: I also tried the queries by splitting them up into multiple parts with an aggregator connection. Still, I have yet to receive any error. I have a few follow-up questions. Please review the following:
Thank You! |
|
Hi @ParthaI I had a look at this and realised I actually shared the wrong query! My apologies there. The query with the issue is: I have retried this today: List of plugins: Steampipe ver: In my plugin.log, logs are filled with errors like this: I am happy to share the full log over slack, but it only contains above errors, nothing much else. I believe if you have two AWS accounts, each containing repos with different names, you should be able to reproduce this. I am sure it would be fixed by using a ParentHydrate similar to #1373 (comment) |
|
Hello @captainfalcon23, I appreciate you providing the information. I'll review the issue once more and let you know if I need anything else. Thank you! |
|
Hi @captainfalcon23, I attempted to reproduce the error and discovered that it's being logged in my LOG file but not appearing in my Terminal. This behavior is expected in aggregator mode because a repository named Could you please confirm you are getting the error in your terminal or in your LOG file only?
Thanks! |
|
Hi @ParthaI
Is there any better solution or way of writing this (and other similar) queries to avoid this situation? I fell back onto doing a query for each and every account individually, and aggregating them at the end, but it's not flexible. I think also this is not a common scenario (I may be wrong) where there are 3 tables in the chain. Happy to discuss and get some suggestions :) |
|
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days. |
github-actions
bot
added
the
stale
ParthaI
removed
the
stale
|
@captainfalcon23, after careful consideration and review of the issue you've raised, we have decided to close this issue at the current time. We understand that the unnecessary logs have been a concern for you, and to address this specifically, I have raised an issue with our SDK team for further investigation and resolution. You can track the progress here. We are committed to ensuring our service meets your needs and expectations, and we value your feedback and contributions to our community. If you have any additional insights or feedback, or if there's anything specific you would like to discuss further or contribute, please do not hesitate to reopen this issue or create a new one. Additionally, if you feel inclined to contribute directly through a Pull Request (PR), we welcome your contributions with open arms. |
|
That’s fine, but the issue isn’t about logs in a log file. It’s about steampipe inefficiency of querying every account and region in an aggregation instead of smartly hydrating the tables from parents. If the context is there that an image belongs to account X in region Y, I don’t understand why steampipe can’t use that context to query the appropriate connection for further info. |
|
@captainfalcon23, the |
|
FYI I haven't forgotten about this, will take a look today |
|
Okay, I have compiled the plugin and ran the same query as before and still I can see errors where steampipe is trying to find images in the wrong accounts. I have simplified the query, and you'll see on the 4th and 5th last lines, there is an additional join condition. But the logs still show Steampipe is checking each and every account for every repository. However, the region filter DOES appear to be working, as there is only a max of 1 call to each account to find the repo, so I think we are half way there! I tested adding/removing the region in the join, and it is clearly using the region and even the query returns much quicker when region is used in the join. Using region + account_id in join: WITHOUT Using region + account_id in join: |
|
@captainfalcon23, Could you please try out this PR, I have tested the above queries and it is working fine for me without any error logs. |
|
@bigdatasourav Doesn't seem to be working as expected for me. This is what I've done: I can still see errors in the logs, indicating the accountid isn't being used in the join. Am I missing a step somewhere? |
|
Hi @misraved Just checking in why this issue got closed off when the issue is not fixed? |
Describe the bug
Pretty similar to #1373.
Trying to join aws_ecr_image onto aws_ecr_image_scan_finding like:
The problem is, to query aws_ecr_image_scan_finding you need to pass repo name + image tag, and when this is done via a join, steampipe is checking each and every aggregated connection for the existence of the repository. When it can't be found, thousands of errors are dumped to the logs:
Steampipe version (
steampipe -v)Steampipe v0.21.1
Plugin version (
steampipe plugin list)To reproduce
Query:
Expected behavior
Only the account where the image exists is queried.
The text was updated successfully, but these errors were encountered: