Fix parsing of row events for MySQL8 partitioned table#355
Merged
julien-duponchelle merged 3 commits intojulien-duponchelle:mainfrom Aug 30, 2021
dongwook-chan:extra-data
Merged
Fix parsing of row events for MySQL8 partitioned table#355julien-duponchelle merged 3 commits intojulien-duponchelle:mainfrom dongwook-chan:extra-data
julien-duponchelle merged 3 commits intojulien-duponchelle:mainfrom
dongwook-chan:extra-data
Conversation
Since MySQL 8.0.16, partition info has been [added](mysql/mysql-server@e11a540) into extra_data in row event. Current code is aware of extra_data but doesn't parse it accurately. Its length is defined as 'extra_data_length / 8'. However, the [maximum length of partition info](https://github.com/mysql/mysql-server/blob/beb865a960b9a8a16cf999c323e46c5b0c67f21f/libbinlogevents/include/rows_event.h#L772-L814) is 5. Wrong definition leads program to read 0 (= 5/8) bytes of extra_data. This causes any row event on partitioned table leads to packet parse failure (undefined behavior). In some cases errors occur, and in other cases parse result shows incorrect filed values as in [issue #354](#354 (comment)). 1. Fix byte-length of extra_data to conform to repl protocol extra_data_length / 8 -> extra_data_length - 2 Refer to [MySQL Document](https://dev.mysql.com/doc/internals/en/rows-event.html) and [MySQL source code]((https://github.com/mysql/mysql-server/blob/beb865a960b9a8a16cf999c323e46c5b0c67f21f/libbinlogevents/src/rows_event.cpp#L415)). Above fixes failure of packet parsing, but following modification is further required to get partition info. 2. Parse extra_data according to [partition info layout](https://github.com/mysql/mysql-server/blob/beb865a960b9a8a16cf999c323e46c5b0c67f21f/libbinlogevents/include/rows_event.h#L772-L814).
test_extra_data -> test_partition_id
dongwook-chan
changed the title
heehehe
added a commit
to python-mysql-replication-kr/python-mysql-replication
that referenced
this pull request
Sep 19, 2023
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since MySQL 8.0.16, partition info has been added into extra_data in row event.
Current code is aware of extra_data but doesn't parse it accurately. The program reads
extra_data_length / 8bytes from packet instead of correct byte-lengthextra_data_length - 2. This causes the library to skip parsing of extra_data.In some cases exceptions are raised, and in other cases parse result shows incorrect field values as in issue #354. (undefined behavior)
below are the modifications made:
Correct byte-length of extra_data to conform to replication protocol
extra_data_length / 8->extra_data_length - 2https://github.com/noplay/python-mysql-replication/blob/3de6ff499f7695a800409341f4f859cac5b724d0/pymysqlreplication/row_event.py#L60
Refer to MySQL Document and MySQL source code for correct bytes-to-read.
Parse extra_data according to extra_data layout to get partition id ( or ndb info)