github / github/gh-ost

Process inserting nulls in almost all fields

Open
#733 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
13.6k
Forks
1.4k
Avg merge
2h 31m
Merged PRs (30d)
4

Description

When trying to run on MySQL 8.0.16 row values do not seem to be correctly inserted. The errors continue for each attempted insert.

```log
db:~/gh-ost# ~/gh-ost/bin/gh-ost --max-load=Threads_running=25 --critical-load=Threads_running=1000 --chunk-size=1000 --throttle-control-replicas="slave" --max-lag-millis=1500 --user="repl" --password="un-mam-zul-nu-fru-ow-a" --host=slave --verbose --exact-rowcount --default-retries=120 --allow-on-master --panic-flag-file=/tmp/ghost.panic.flag --database="osu" --table="phpbb_users" --cut-over=two-step --alter="modify user_email varchar(100) COLLATE utf8_general_ci DEFAULT NULL" --execute --initially-drop-ghost-table
2019-05-01 01:57:15 INFO starting gh-ost ffc6c40150efb149321a2f32103b98424d6652b1
2019-05-01 01:57:15 INFO Migrating `osu`.`phpbb_users`
2019-05-01 01:57:15 INFO connection validated on slave:3306
2019-05-01 01:57:15 INFO User has SUPER, REPLICATION SLAVE privileges, and has ALL privileges on `osu`.*
2019-05-01 01:57:15 INFO binary logs validated on slave:3306
2019-05-01 01:57:15 INFO Restarting replication on slave:3306 to make sure binlog settings apply to replication thread
2019-05-01 01:57:15 INFO Inspector initiated on db:3306, version 8.0.16
2019-05-01 01:57:15 INFO Table found. Engine=InnoDB
2019-05-01 01:57:16 INFO Estimated number of rows via EXPLAIN: 11940607
2019-05-01 01:57:16 INFO Recursively searching for replication master
2019-05-01 01:57:16 INFO Master found to be db:3306
2019-05-01 01:57:16 INFO log_slave_updates validated on slave:3306
2019-05-01 01:57:16 INFO connection validated on slave:3306
2019-05-01 01:57:16 INFO Connecting binlog streamer at mysql-bin.007350:747779741
[2019/05/01 01:57:16] [info] binlogsyncer.go:133 create BinlogSyncer with config {99999 mysql slave 3306 repl false false false UTC true 0 0s 0s 0 false}
[2019/05/01 01:57:16] [info] binlogsyncer.go:354 begin to sync binlog from position (mysql-bin.007350, 747779741)
[2019/05/01 01:57:16] [info] binlogsyncer.go:203 register slave for master server slave:3306
[2019/05/01 01:57:16] [info] binlogsyncer.go:723 rotate to (mysql-bin.007350, 747779741)
2019-05-01 01:57:16 INFO connection validated on slave:3306
2019-05-01 01:57:16 INFO rotate to next log from mysql-bin.007350:0 to mysql-bin.007350
2019-05-01 01:57:16 INFO connection validated on slave:3306
2019-05-01 01:57:16 INFO will use time_zone='SYSTEM' on applier
2019-05-01 01:57:16 INFO Examining table structure on applier
2019-05-01 01:57:16 INFO Applier initiated on db:3306, version 8.0.16
2019-05-01 01:57:16 INFO Dropping table `osu`.`_phpbb_users_gho`
2019-05-01 01:57:16 INFO Table dropped
2019-05-01 01:57:16 INFO Dropping table `osu`.`_phpbb_users_ghc`
2019-05-01 01:57:16 INFO Table dropped
2019-05-01 01:57:16 INFO Creating changelog table `osu`.`_phpbb_users_ghc`
2019-05-01 01:57:16 INFO Changelog table created
2019-05-01 01:57:16 INFO Creating ghost table `osu`.`_phpbb_users_gho`
2019-05-01 01:57:16 INFO Ghost table created
2019-05-01 01:57:16 INFO Altering ghost table `osu`.`_phpbb_users_gho`
2019-05-01 01:57:16 INFO Ghost table altered
2019-05-01 01:57:16 INFO Intercepted changelog state GhostTableMigrated
2019-05-01 01:57:16 INFO Waiting for ghost table to be migrated. Current lag is 0s
2019-05-01 01:57:16 INFO Handled changelog state GhostTableMigrated
2019-05-01 01:57:16 INFO Chosen shared unique key is PRIMARY
2019-05-01 01:57:16 INFO Shared columns are user_id,user_type,group_id,user_permissions,user_perm_from,user_ip,user_regdate,username,username_clean,user_password,user_passchg,user_email,user_birthday,user_lastvisit,user_lastmark,user_lastpost_time,user_lastpage,user_last_confirm_key,user_last_search,user_warnings,user_last_warning,user_login_attempts,user_inactive_reason,user_inactive_time,user_posts,user_lang,user_timezone,user_dst,user_dateformat,user_style,user_rank,user_colour,user_new_privmsg,user_unread_privmsg,user_last_privmsg,user_message_rules,user_full_folder,user_emailtime,user_topic_show_days,user_topic_sortby_type,user_topic_sortby_dir,user_post_show_days,user_post_sortby_type,user_post_sortby_dir,user_notify,user_notify_pm,user_notify_type,user_allow_pm,user_allow_viewonline,user_allow_viewemail,user_allow_massemail,user_options,user_avatar,user_avatar_type,user_avatar_width,user_avatar_height,user_sig,user_sig_bbcode_uid,user_sig_bbcode_bitfield,user_from,user_lastfm,user_lastfm_session,user_twitter,user_msnm,user_jabber,user_website,user_occ,user_interests,user_actkey,user_newpasswd,osu_mapperrank,osu_testversion,osu_subscriber,osu_subscriptionexpiry,osu_kudosavailable,osu_kudosdenied,osu_kudostotal,country_acronym,userpage_post_id,username_previous,osu_featurevotes,osu_playstyle,osu_playmode,remember_token
2019-05-01 01:57:16 INFO Listening on unix socket file: /tmp/gh-ost.osu.phpbb_users.sock
2019-05-01 01:57:16 INFO As instructed, counting rows in the background; meanwhile I will use an estimated count, and will update it later on
2019-05-01 01:57:16 INFO As instructed, I'm issuing a SELECT COUNT(*) on the table. This may take a while
2019-05-01 01:57:16 INFO Migration min values: [1]
2019-05-01 01:57:16 INFO Migration max values: [14376598]
2019-05-01 01:57:16 INFO Waiting for first throttle metrics to be collected
2019-05-01 01:57:16 INFO First throttle metrics collected
# Migrating `osu`.`phpbb_users`; Ghost table is `osu`.`_phpbb_users_gho`
# Migrating db:3306; inspecting db:3306; executing on db
# Migration started at Wed May 01 01:57:15 +0000 2019
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 10; max-load: Threads_running=25; critical-load: Threads_running=1000; nice-ratio: 0.000000
# throttle-additional-flag-file: /tmp/gh-ost.throttle
# throttle-control-replicas count: 1
# panic-flag-file: /tmp/ghost.panic.flag
# Serving on unix socket: /tmp/gh-ost.osu.phpbb_users.sock
Copy: 0/11940607 0.0%; Applied: 0; Backlog: 0/1000; Time: 0s(total), 0s(copy); streamer: mysql-bin.007350:747970447; State: migrating; ETA: N/A
Copy: 0/11940607 0.0%; Applied: 0; Backlog: 25/1000; Time: 1s(total), 1s(copy); streamer: mysql-bin.007350:748319292; State: migrating; ETA: N/A
2019-05-01 01:57:17 ERROR Error 1048: Column 'user_type' cannot be null; query=
replace /* gh-ost `osu`.`_phpbb_users_gho` */ into
`osu`.`_phpbb_users_gho`
(`user_id`, `user_type`, `group_id`, `user_permissions`, `user_perm_from`, `user_ip`, `user_regdate`, `username`, `username_clean`, `user_password`, `user_passchg`, `user_email`, `user_birthday`, `user_lastvisit`, `user_lastmark`, `user_lastpost_time`, `user_lastpage`, `user_last_confirm_key`, `user_last_search`, `user_warnings`, `user_last_warning`, `user_login_attempts`, `user_inactive_reason`, `user_inactive_time`, `user_posts`, `user_lang`, `user_timezone`, `user_dst`, `user_dateformat`, `user_style`, `user_rank`, `user_colour`, `user_new_privmsg`, `user_unread_privmsg`, `user_last_privmsg`, `user_message_rules`, `user_full_folder`, `user_emailtime`, `user_topic_show_days`, `user_topic_sortby_type`, `user_topic_sortby_dir`, `user_post_show_days`, `user_post_sortby_type`, `user_post_sortby_dir`, `user_notify`, `user_notify_pm`, `user_notify_type`, `user_allow_pm`, `user_allow_viewonline`, `user_allow_viewemail`, `user_allow_massemail`, `user_options`, `user_avatar`, `user_avatar_type`, `user_avatar_width`, `user_avatar_height`, `user_sig`, `user_sig_bbcode_uid`, `user_sig_bbcode_bitfield`, `user_from`, `user_lastfm`, `user_lastfm_session`, `user_twitter`, `user_msnm`, `user_jabber`, `user_website`, `user_occ`, `user_interests`, `user_actkey`, `user_newpasswd`, `osu_mapperrank`, `osu_testversion`, `osu_subscriber`, `osu_subscriptionexpiry`, `osu_kudosavailable`, `osu_kudosdenied`, `osu_kudostotal`, `country_acronym`, `userpage_post_id`, `username_previous`, `osu_featurevotes`, `osu_playstyle`, `osu_playmode`, `remember_token`)
values
(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
; args=[ 1556675820 ]
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.