The height of bounding boxes for some words is much larger than the words

This is causing some problems when determining the midpoints for the bounding boxes for the fully convolutional network. It's basically putting the midpoint in a different spot then it would have been had the bounding box tightly enclosed the text. Any ideas?