Unexpected behavior with geom_text layers in ggplotly

I'm getting unexpected behavior when assigning color to geom_text() in ggplotly

This is slightly related to issue #469 (and I'll use the same example data) where a geom_text layer wasn't appearing at all. That's no longer the case. The text appears fine (yeah!) but the interactivity isn't functioning as I would expect (rats!).

Here's some data:

tempDf <- data.frame("x" = c(22.1, 22.1, 14.4, 32.1, 22.0, 32.1, 84.3, 15.9, 15.9, 11.8, 11.8, 25.1, 15.7, 22.4, 35.1, 36.6, 70.4, 65.2, 42.9, 36.3, 47.8),
"y" = c(21.0, 21.0, 39.8, 47.6, 69.6, 95.0, 94.8, 17.3, 17.3, 27.6, 27.6, 90.3, 85.7, 94.1, 72.3, 71.5, 100.0, 92.4, 76.9, 34.6, 4.9),
"endX" = c(NA, 13.1, 31.9, 22.0, 29.7, 84.3, 94.4, NA, 15.1, NA, 25.0, 14.3, 22.4, 59.6, NA, 70.4, 65.9, 44.2, 38.3, 48.3, 72.1),
"endY" = c(NA, 33.3, 46.0, 69.6, 93.5, 94.8, 51.6, NA, 22.1, NA, 90.3, 85.4, 94.1, 96.8, NA, 100.0, 91.5, 80.8, 37.1, 5.2, 36.9),
"index" = c(1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0),
"string"= c(1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0) )

And here's my plot:

example <- ggplot(tempDf, aes(x=x, y=y, color=as.factor(index))) + geom_point(size = 4, shape = 0) + 
    geom_text(aes(label= string), size = 2) + 
    geom_segment(data = tempDf, aes(x=x, xend=endX, y=y, yend=endY, group = index, color=as.factor(index)), size = .3, arrow=arrow(length=unit(0.3,"cm")) ) + xlim(0,100) + ylim(0,100) +
    theme(panel.grid.minor = element_blank(), 
        panel.grid.major = element_blank(), 
        panel.background = element_rect(fill = "grey90"), 
        axis.ticks.x = element_blank(), 
        axis.ticks.y = element_blank(), 
        axis.text.x = element_blank(), 
        axis.text.y = element_blank(),  
        #legend.title=element_blank(),
        #legend.margin=unit(-.4, "cm"),
        plot.margin=unit(c(0, 0, 1.3, -1), units="cm"),  #top, right, bottom, left
        legend.text = element_text(size = 14, face = "bold") ) + 
    xlab("") + ylab("")
ggplotly(example)

There are three colored sequences (at least that's how I'll refer to them here, one red, one green and one blue). I can toggle off/on ant of the individual color sequences. When I do that all of the elements toggle off (the segments, the text (number) and the shape; arrows aren't yet supported in plotly). Here, I've toggled the red off. This is as I would expect (and what I'm after).

However, if I change the color of the geom_text like this (the only difference is that I'm adding: 'color = "black"' to the geom_text layer)

example <- ggplot(tempDf, aes(x=x, y=y, color=as.factor(index))) + geom_point(size = 4, shape = 0) + 
        geom_text(aes(label= string), size = 2, color = "black") + 
        geom_segment(data = tempDf, aes(x=x, xend=endX, y=y, yend=endY, group = index, color=as.factor(index)), size = .3, arrow=arrow(length=unit(0.3,"cm")) ) + xlim(0,100) + ylim(0,100) +
        theme(panel.grid.minor = element_blank(), 
            panel.grid.major = element_blank(), 
            panel.background = element_rect(fill = "grey90"), 
            axis.ticks.x = element_blank(), 
            axis.ticks.y = element_blank(), 
            axis.text.x = element_blank(), 
            axis.text.y = element_blank(),  
            #legend.title=element_blank(),
            #legend.margin=unit(-.4, "cm"),
            plot.margin=unit(c(0, 0, 1.3, -1), units="cm"),  #top, right, bottom, left
            legend.text = element_text(size = 14, face = "bold") ) + 
        xlab("") + ylab("")
ggplotly(example)

Now when I toggle off any of the colored sequences, the black numbers are left behind. Here I've left the red layer 'on') but notice all of the black letters from the other sequences are left still showing.

That's not right, is it? The behavior I'd expect (and what I'm hoping for) is that the black numbers also toggle off with the rest of the elements.

Start by reproducing the supplied ggplotly example with geom_point, geom_text, and geom_segment, comparing mapped color with a fixed black text color. Trace how ggplotly groups layers for legend toggling. Done means black geom_text labels toggle off with the corresponding colored sequence, while the existing behavior remains intact.