This is an automated comment for commit 424cd1b with description of existing statuses. It's updated for the latest CI running

✅ Click here to open a full report in a separate page

One more bug, need to check if monotonic_functions_chain is empty.

Fairly interesting. I just tested it and it would indeed be a problem. It is a problem for min/max stats evaluation as well:

arthur :) select * from file('afile.parquet', Parquet) where negate(int8_logical) = -58;

SELECT *
FROM file('afile.parquet', Parquet)
WHERE (-int8_logical) = -58

Query id: 7e350ea0-0e79-4cc6-867a-b58b06659f52

Ok.

0 rows in set. Elapsed: 15.389 sec. 

arthur :) select * from file('afile.parquet', Parquet) where negate(int8_logical) = -58 SETTINGS input_format_parquet_filter_push_down=false;

SELECT *
FROM file('afile.parquet', Parquet)
WHERE (-int8_logical) = -58
SETTINGS input_format_parquet_filter_push_down = false

Query id: 3bff6c4b-92ca-4f06-b2e9-1829a8c7873f

   ┌─int8_logical─┬─uint16_logical─┬─ipv4─┬──────int64─┬──────uint64_logical─┬─boolean─┬─string─┬─json─────────────────────────┬─flba─┬─uuid────┐
1. │           58 │          57159 │  631 │ 4294967365 │ 9829797005067372154 │ true    │ RRBT   │ {"key":291, "value":"DUBPO"} │ NDBX │ �G�)@��譔�:� │
   └──────────────┴────────────────┴──────┴────────────┴─────────────────────┴─────────┴────────┴──────────────────────────────┴──────┴─────────┘
   ┌─int8_logical─┬─uint16_logical─┬─ipv4─┬──────int64─┬───────uint64_logical─┬─boolean─┬─string─┬─json─────────────────────────┬─flba─┬─uuid──────┐
2. │           58 │          63254 │  986 │ 4294967373 │ 12146296632881510326 │ false   │ SOOF   │ {"key":244, "value":"XPHCW"} │ BBET │ XH�rBr�,����SN  │
3. │           58 │          14652 │  334 │ 4294967346 │ 16224854854285702292 │ false   │ MWPY   │ {"key":592, "value":"NJOJN"} │ PADH │ L^�e�
                                                                                                                                              A;�� Y�f�, │
   └──────────────┴────────────────┴──────┴────────────┴──────────────────────┴─────────┴────────┴──────────────────────────────┴──────┴───────────┘
   ┌─int8_logical─┬─uint16_logical─┬─ipv4─┬──────int64─┬──────uint64_logical─┬─boolean─┬─string─┬─json─────────────────────────┬─flba─┬─uuid────┐
4. │           58 │          29205 │  586 │ 4294967321 │ 4503168388465453601 │ true    │ CDTD   │ {"key":208, "value":"LOCNZ"} │ UVDZ │ 4��|B���zo?�" │
�B�� │         58 │          22488 │   75 │ 4294967378 │ 1882125239624409868 │ false   │ NGRD   │ {"key":22, "value":"OVAPX"}  │ CDOA │ Cn��pO�� 
   └──────────────┴────────────────┴──────┴────────────┴─────────────────────┴─────────┴────────┴──────────────────────────────┴──────┴─────────┘

5 rows in set. Elapsed: 2.629 sec. Processed 1.00 thousand rows, 47.42 KB (380.38 rows/s., 18.04 KB/s.)
Peak memory usage: 129.99 KiB.

It is a problem for min/max stats evaluation as well:

I couldn't reproduce. What's in afile.parquet?

insert into function file('t.parquet') select 42::Int64 as x settings engine_file_truncate_on_insert=1;

select * from file('t.parquet') where negate(x) = -42 settings input_format_parquet_filter_push_down=1

SELECT *
FROM file('t.parquet')
WHERE (-x) = -42
SETTINGS input_format_parquet_filter_push_down = 1

Query id: 472f9083-ac38-4aa5-9cc4-2191f45639ca

   ┌──x─┐
1. │ 42 │
   └────┘

1 row in set. Elapsed: 0.003 sec. 

-- check that index is used at all:
select * from file('t.parquet') where indexHint(negate(x) = -43) settings input_format_parquet_filter_push_down=1

SELECT *
FROM file('t.parquet')
WHERE indexHint((-x) = -43)
SETTINGS input_format_parquet_filter_push_down = 1

Query id: 3f2e7936-ec56-431d-a399-c0e6914257d4

Ok.

0 rows in set. Elapsed: 0.003 sec. 

Min/max thing shouldn't have this exact problem because it calls KeyCondition::checkInHyperrectangle, which applies the monotonic functions.

It is a problem for min/max stats evaluation as well:

I couldn't reproduce. What's in afile.parquet?

insert into function file('t.parquet') select 42::Int64 as x settings engine_file_truncate_on_insert=1;



select * from file('t.parquet') where negate(x) = -42 settings input_format_parquet_filter_push_down=1



SELECT *

FROM file('t.parquet')

WHERE (-x) = -42

SETTINGS input_format_parquet_filter_push_down = 1



Query id: 472f9083-ac38-4aa5-9cc4-2191f45639ca



   ┌──x─┐

1. │ 42 │

   └────┘



1 row in set. Elapsed: 0.003 sec. 



-- check that index is used at all:

select * from file('t.parquet') where indexHint(negate(x) = -43) settings input_format_parquet_filter_push_down=1



SELECT *

FROM file('t.parquet')

WHERE indexHint((-x) = -43)

SETTINGS input_format_parquet_filter_push_down = 1



Query id: 3f2e7936-ec56-431d-a399-c0e6914257d4



Ok.



0 rows in set. Elapsed: 0.003 sec. 

Min/max thing shouldn't have this exact problem because it calls KeyCondition::checkInHyperrectangle, which applies the monotonic functions.

It should be the same as the file in this PR. Maybe I missed something while testing, will double check tomorrow.

It is a problem for min/max stats evaluation as well:

I couldn't reproduce. What's in afile.parquet?

insert into function file('t.parquet') select 42::Int64 as x settings engine_file_truncate_on_insert=1;

select * from file('t.parquet') where negate(x) = -42 settings input_format_parquet_filter_push_down=1

SELECT *
FROM file('t.parquet')
WHERE (-x) = -42
SETTINGS input_format_parquet_filter_push_down = 1

Query id: 472f9083-ac38-4aa5-9cc4-2191f45639ca

   ┌──x─┐
1. │ 42 │
   └────┘

1 row in set. Elapsed: 0.003 sec. 

-- check that index is used at all:
select * from file('t.parquet') where indexHint(negate(x) = -43) settings input_format_parquet_filter_push_down=1

SELECT *
FROM file('t.parquet')
WHERE indexHint((-x) = -43)
SETTINGS input_format_parquet_filter_push_down = 1

Query id: 3f2e7936-ec56-431d-a399-c0e6914257d4

Ok.

0 rows in set. Elapsed: 0.003 sec. 

Min/max thing shouldn't have this exact problem because it calls KeyCondition::checkInHyperrectangle, which applies the monotonic functions.

Running this query with bloom_filter = false and minmax_filter = true returns inconsistent results

Minimized to:

insert into function file('t.parquet') select 12345678901234567890::UInt64 as x union all select 42::UInt64 as x settings engine_file_truncate_on_insert=1;
select * from file('t.parquet') where -x = -42;

The problem is that negate overflows on the max value from statistics, and KeyCondition doesn't notice and ends up with inverted range. Fix: #71440