Opensearch datasource not working properly with data.timestamp field in grafana

[SricharanSundar]

What happened:
The time field is not properly mapped with grafana time series visualization. I written the lucene query in grafana to plot a time and counts based on data.timestamp field. But the visualization is working for Time field of ingestion field.
What you expected to happen:
The time series visualization need to work as per the data.timestamp field and dont need to work on Time field in opensearch.
How to reproduce it (as minimally and precisely as possible):
Ingest some logs according the following criteria and check. The timestamp field values are not organized in desc or asc its mixed like the following way.
Time data.timestamp
Jul 22, 2024 @ 23:54:23.433 Jul 22, 2024 @ 23:45:27.784
Jul 22, 2024 @ 23:54:23.177 Jul 22, 2024 @ 23:45:31.602
Jul 22, 2024 @ 23:54:19.967 Jul 22, 2024 @ 23:47:58.090
Jul 22, 2024 @ 23:54:17.367 Jul 22, 2024 @ 23:50:17.581
Jul 22, 2024 @ 23:54:17.108 Jul 22, 2024 @ 23:50:40.333
Jul 22, 2024 @ 23:54:16.365 Jul 22, 2024 @ 23:51:13.119
Jul 22, 2024 @ 23:54:14.256 Jul 22, 2024 @ 23:52:02.211
Jul 22, 2024 @ 23:54:12.766 Jul 22, 2024 @ 23:53:13.873
Anything else we need to know?:
I need to create a time series visualization using the data.timestamp with counts.
Current issue: If i mention the data histogram as data.timestamp also its working by default for Time field in grafana.

Environment: Prod

• Grafana version: 10.4.1
• OpenSearch version: 2.11
• Plugin version: 2.14.7
• Plugin API version: grafana-opensearch-datasource/versions/2.14.7/download

[kevinwcyu]

Hi @SricharanSundar, just wanted to clarify the problem. You have logs where there are two fields that are time data types, timestamp and Time, where timestamp is the field you want the visualization to use and Time is a field that represents when the log was ingested? Is that correct?

What do you have set for the Time field name in your OpenSearch data source configuration?

[SricharanSundar]

Yes, that is correct.

…

On Wed, 24 Jul 2024 at 11:48 PM, Kevin Yu ***@***.***> wrote: Hi @SricharanSundar <https://github.com/SricharanSundar>, just wanted to clarify the problem. You have logs where there are two fields that are time data types, timestamp and Time, where timestamp is the field you want the visualization to use and Time is a field that represents when the log was ingested? Is that correct? What do you have set for the Time field name in your OpenSearch data source configuration? — Reply to this email directly, view it on GitHub <#425 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZ2A5EAJBA4OSSX7G4JUU5LZN7VWLAVCNFSM6AAAAABLJ6ARSCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBYGYZTSMJXGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[kevinwcyu]

Hi @SricharanSundar,

What do you have set for the Time field name in your OpenSearch data source configuration?

[SricharanSundar]

Hi @SricharanSundar,

What do you have set for the Time field name in your OpenSearch data source configuration?

"data": {
"properties": {
"aggregate_id": {
"type": "text"
},
"container_id": {
"type": "keyword"
},
"data_center": {
"type": "keyword"
},
"timestamp": {
"type": "date_nanos"
},

[iwysiu]

Hi @SricharanSundar ! In the datasource configuration in Grafana, do you have Time field name set to data.timestamp? Below is a picture where the time field is set to order_date.