Skip to content
Abhi Basu edited this page Jun 5, 2015 · 8 revisions

mysql_etl.py

Dealing with LONGBLOB fields when using Sqoop to import from MySQL to Hive.

We were attempting to export tables out of MySQL to Hive and ran into an issue where this particular table had LONGBLOB column type which Sqoop did not care for :).

So this is what we did:

  1. Create a new TEXT column
  2. Convert contents of LONGBLOB column into text (utf8), save in new column
  3. Delete old column.

Dependency: https://github.com/PyMySQL/PyMySQL (pymysql connector)

Clone this wiki locally