Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PARQUET-675: Add INTERVAL_YEAR_MONTH and INTERVAL_DAY_TIME types #43

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

PARQUET-675: Add INTERVAL_YEAR_MONTH and INTERVAL_DAY_TIME types #43

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e39aa7b
PARQUET-675: Add INTERVAL_YEAR_MONTH and INTERVAL_DAY_TIME types
julienledem Oct 26, 2016
ddc9c47
add LogicalTypes description
julienledem Oct 28, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 24 additions & 1 deletion LogicalTypes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,8 +130,12 @@ milliseconds from the Unix epoch, 00:00:00.000 on 1 January 1970, UTC.
microsecond precision. It must annotate an `int64` that stores the number of
microseconds from the Unix epoch, 00:00:00.000000 on 1 January 1970, UTC.

### INTERVAL
### INTERVAL types

#### INTERVAL
`INTERVAL` is deprecated. Please use the more precise `INTERVAL_YEAR_MONTH` and
`INTERVAL_DAY_TIME` per SQL spec.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we link to the SQL spec here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

http://webstore.ansi.org/RecordDetail.aspx?sku=ISO%2fIEC+9075-1%3a2011

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I meant if we could add a link to the sql spec in the md file, it might be useful for folks reading in the future.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I should have added a comment. The SQL spec is not free. This is a paid link.
There are drafts hosted here and there. We can use those.

For historical reference:
`INTERVAL` is used for an interval of time. It must annotate a
`fixed_len_byte_array` of length 12. This array stores three little-endian
unsigned integers that represent durations at different granularities of time.
Expand All @@ -144,6 +148,25 @@ example, there is no requirement that a large number of days should be
expressed as a mix of months and days because there is not a constant
conversion from days to months.

#### INTERVAL_YEAR_MONTH
`INTERVAL_YEAR_MONTH` describes an interval of time with year-month unit.
It annotates data stored as an INT32
This data is stored as a little endian unsigned
integer identifying the number of months associated with the duration.
This duration of time is independent of any
particular timezone or date.

#### INTERVAL_DAY_TIME
`INTERVAL_DAY_TIME` describes interval of time with days-milliseconds unit
This type annotates data stored as a FIXED_LEN_BYTE_ARRAY of length 8
This data is composed of two separate little endian unsigned
integers. Each stores a component of a duration of time.

The first identifies the number of days associated with the duration
and the second identifies the number of milliseconds associated with
the provided duration. This duration of time is independent of any
particular timezone or date.

## Embedded Types

### JSON
Expand Down
35 changes: 32 additions & 3 deletions src/main/thrift/parquet.thrift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,19 +162,48 @@ enum ConvertedType {
BSON = 20;

/**
* @Deprecated: use INTERVAL_YEAR_MONTH or INTERVAL_DAY_TIME
* since the SQL standard defines either YEAR_MONTH or DAY_TIME unit.
* This is deprecated in favor of those 2 types
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does the SQL spec say?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See links in https://issues.apache.org/jira/browse/PARQUET-675.
For example: "An interval is defined as the difference between two dates and times. Intervals are expressed in one of two different ways. One is a year-month interval that expresses intervals in terms of years and an integral number of months. The other is a day-time interval that expresses intervals in terms of days, minutes, and seconds. These two types of intervals are distinct and cannot be mixed, because months can have varying numbers of days."
https://msdn.microsoft.com/en-us/library/ms716506(v=vs.85).aspx

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I add this quote in there?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SQL 92 Spec:

     4.5.2  Intervals
     There are two classes of intervals. One class, called year-month
     intervals, has an express or implied datetime precision that in-
     cludes no fields other than YEAR and MONTH, though not both are
     required. The other class, called day-time intervals, has an ex-
     press or implied interval precision that can include any fields
     other than YEAR or MONTH.

Further:

     Operations involving items of
     type interval require that the interval items be mutually compara-
     ble.
     ...
     Year-month intervals are mutually comparable only with other year-
     month intervals
     ...
     Day-time intervals are mutually comparable only with other day-
     time intervals.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How are intervals comparable? Are there details in the spec for resolving cases like (1 day, 0 ms) <=> (0 days, 86400000 ms)?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the SQL definition 1 day = 24 hours = 86400 seconds.

     _Keyword______Valid_values_of_INTERVAL_fields______________________
    | DAY        | Unconstrained except by <interval leading field     |
                   precision

    | HOUR       | Hours (within days) (0-23)                          |
    |            |                                                     |
    | MINUTE     | Minutes (within hours) (0-59)                       |
    |            |                                                     |
    |_SECOND_____|_Seconds_(within_minutes)_(0-59.999...)______________|

and

     When it is
     necessary to add new most significant datetime fields, the asso-
     ciated value is effectively converted to the new precision in a
     manner obeying the natural rules for dates and times associated
     with the Gregorian calendar.

Of course this is just the standard SQL, Parquet does not have to follow this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@michal-databricks, can you post the reference for 1 day = 24 hours = 86400 seconds? I'm not sure that what you've quoted here requires it. The second paragraph, "When it is necessary...", is talking about conversion and the first constrains the fields. That doesn't mean that "1 day" and "23 hours 59 minutes 59.999999 seconds" are comparable. It just means that 86400 seconds is an invalid interval.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rdblue, this is the complete paragraph defining the comparison of day-time intervals where the quote was taken from.

     Day-time intervals are mutually comparable only with other day-
     time intervals. If two day-time intervals have different interval
     precisions, they are, for the purpose of any operations between
     them, effectively converted to the same precision by appending new
     <datetime field>s to either the most significant end or the least
     significant end of one or both day-time intervals. New least sig-
     nificant <datetime field>s are assigned a value of 0. When it is
     necessary to add new most significant datetime fields, the asso-
     ciated value is effectively converted to the new precision in a
     manner obeying the natural rules for dates and times associated
     with the Gregorian calendar.

To my understanding, this means that, for example, comparing INTERVAL '100' HOUR with INTERVAL '4 04:00' DAY TO MINUTE requires conversion to the common super type, in this case INTERVAL DAY TO MINUTE. During conversion least significant fields are filled with zeros and most significant "in a manner obeying the natural rules for dates and times associated with the Gregorian calendar", since the non-most-significant fields must conform to the constraints in the table (comes from the same chapter). In this case INTERVAL '100' HOUR becomes INTERVAL '4 04:00' DAY TO MINUTE. Now they are compared field by field (making them equal).
This is the best reference from the SQL '92 standard I can give to argument 1 day = 24 hours. And while implementations of the standard vary they still tend to use the above method for comparisons.

*
* An interval of time
*
*
* This type annotates data stored as a FIXED_LEN_BYTE_ARRAY of length 12
* This data is composed of three separate little endian unsigned
* integers. Each stores a component of a duration of time. The first
* integer identifies the number of months associated with the duration,
* the second identifies the number of days associated with the duration
* and the third identifies the number of milliseconds associated with
* and the third identifies the number of milliseconds associated with
* the provided duration. This duration of time is independent of any
* particular timezone or date.
*/
INTERVAL = 21;


/**
* An interval of time with a year-month unit.
*
* This type annotates data stored as an INT32
* This data is stored as a little endian unsigned
Copy link

@michal-databricks michal-databricks Aug 9, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be signed. SQL standard does not mention this explicitly, but defines interval - interval operation, which would be problematic without negative intervals. Also all SQL implementations I am aware of, allow negative intervals.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

* integer identifying the number of months associated with the duration.
* This duration of time is independent of any
* particular timezone or date.
*/
INTERVAL_YEAR_MONTH = 22;

/**
* An interval of time with days-milliseconds unit
*
* This type annotates data stored as a FIXED_LEN_BYTE_ARRAY of length 8
* This data is composed of two separate little endian unsigned
* integers. Each stores a component of a duration of time.
*
* The first identifies the number of days associated with the duration
* and the second identifies the number of milliseconds associated with
* the provided duration. This duration of time is independent of any
* particular timezone or date.
*/
INTERVAL_DAY_TIME = 23;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there an alternative to embedding two values in a single byte array? The problem is that this is incompatible with stats filtering and efficient integer encodings. I think those are significant enough that we should consider alternatives.

I like the idea of using this annotation for a group of two int32 values. That way we get good compression and encoding, stats filtering works, and we get can get independent dictionary encoding, which would work well for the days field. What are the downsides to that approach instead of the byte array?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the downside is the two values not being contiguous for processing.
We can also make it an Int64 with the most significant 32 bits for day and the rest for ms with the constraint that ms are < 1 day. That would make it sortable.

Copy link
Contributor

@jacques-n jacques-n Oct 28, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that a consumer would be unlikely to read/consume the two values separately, storing them separately seems like overkill. Since we already do the set of three values in the existing interval, it seems like storing two in a single 8 bytes would be okay and consistent.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is that using a single INT64 with milliseconds (1 day = 86400000 milliseconds) would be the most convenient solution. Besides better encoding and statistics it has the advantage of unique representation (does not allow for 1 day and 86400000 ms). Non unique representation would be especially confusing with negative intervals. Single integer storage would be also consistent with the existing storage format of TIMESTAMPs and the storage format for INTERVAL_YEAR_MONTH proposed here.
On that note, it would be better to have 2 types instead: INTERVAL_DAY_MILLIS and INTERVAL_DAY_MICROS just as there are TIMESTAMP_MILLIS and TIMESTAMP_MICROS

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with Jacques that this is lossy because a day isn't consistently 86400000 ms.

We need to store 2 values, but I'm not convinced that there is much value in storing these in a single byte array. What about users that only use days or only use micros? The proposed type would use 8 bytes per value either way. I don't think it is worth that cost when we can deserialize two columns to a single arrow column instead. Is there a micro-benchmark to back up the benefit of using this format instead of a 2-column format?

Using an int64 won't work because with 2 values that aren't constrained by one another, min and max stats are invalid (days always overrides micros). In fact, because days and micros can't be converted to one another, I'd argue that there is no consistent ordering for this interval type. So we would get compression for the days value but at the cost of more confusion about how to produce stats.

Copy link

@michal-databricks michal-databricks Aug 14, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am guessing that by day not equal 86400000 you mean the influence of leap seconds. I don't know any database system which would implement those, especially for intervals. Most just store one component for days and seconds (among them Spark) and while for example Postgres uses separate components for them, it still makes a day equal 24 hours for comparison purposes. But if you do decide to store milliseconds with intention of it being an independent dimension, 32 bits seem not enough: 2 ^ 32 / 86400000 = 49.71. This is only +/-25 "days" range in the signed case.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, another reason you may mean for different day lengths is if they span different DST time zones (on the DST change dates). SQL does not provide a meaningful way to handle those as SQL time stamps only carry time zone offset (in minutes) and not the time zone in the traditional sense (location).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The case I was thinking about was DST. The length of a day may be 23 hours, 24 hours, or 25 hours. That makes 1 day incomparable with 24 hours, unless the SQL spec includes a way to compare them. I'm fine with not having a comparison for Parquet, even if the SQL spec defines one.

The other consideration is whether we should constrain hours, minutes, and seconds to less than 24 hours. The other comment from the SQL spec appears to support that, but constraining the microseconds field to < 86,400,000 will require conversion logic somewhere in order to support arithmetic on interval values: 0 days, 22 hours + 0 days 4 hours = ???. Parquet could just require the microseconds field to be less than 86,400,000, but that shifts the problem to processing engines. And either way, it doesn't support the idea of storing the entire interval using microseconds.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rdblue, there are basically two approaches:

  1. Treat days and seconds as two dimensions. In this case seconds cannot be constrained by 86,400 (otherwise you cannot express 25 hour days on a DST change). Addition and multiplication by a scalar works per dimension. 0 days, 22 hours + 0 days, 4 hours = 0 days, 26 hours.
  2. Treat one day as 24 hours making this one dimension: a scalar of seconds (which I consider the SQL way, but it seems you are not convinced that this is the SQL standard semantic). 0 days 22 hours + 0 days 4 hour = 1 day 2 hours. In this case the storage can be days + 86400000 + ms to make the ordering and arithmetic easier. While it could be still stored as 2 values, it would complicate the implementation and hamper the encoding. Which was my original point.

Going forward with option 1) is fine (as long as it is clearly documented), but then I don't see the advantage over the existing interval type with this is deprecating.


}

/**
Expand Down