Jackson emitting Jaxby @XmAttribute annotated property as an element when a bean serializer modifier is added.

When I register a custom bean serializer modifier with a mapper (version 2.9.6), like so:
```
final SimpleModule module = new SimpleModule();
module.setSerializerModifier(new CustomSerializerModifier(serializationConfig));
xmlMapper.registerModule(module);
```

And also register the Jaxby module (or set annotation introspector ... have tried multiple means).
```
xmlMapper.registerModule(new JaxbAnnotationModule());
```
I get an output that looks like this:
```

0
0
... other elements

```
instead of the expected:
```

... other elements
"
```

Class defined as:

```
/**
* XMLAnnotated class for testing serialization.
*/
@XmlRootElement
public class XmlAnnotated {

private String custName;
private int custAge;
private int custId;
private Element anyElement;

/**
* getCustName.
*/
public String getCustName() {
return custName;
}

/**
* getCustname.
*/
@XmlElement(name = "UserName", required = true)
public void setCustName(final String custName) {
this.custName = custName;
}

/**
* getCustAge.
*/
@XmlAttribute(name = "CustomerAge", required = true)
public int getCustAge() {
return custAge;
}

/**
* getCustAge.
*/
@XmlElement
public void setCustAge(final int custAge) {
this.custAge = custAge;
}

/**
* getCustId.
*/
@XmlAttribute(name = "CustomerId", required = true)
public int getCustId() {
return custId;
}

/**
* setCustId.
*/
@XmlAttribute
public void setCustId(final int custId) {
this.custId = custId;
}

/**
* Gets the Any element parent container.
*/
public Element getAnyElement() {
return anyElement;
}

/**
* Set the Any element.
*/
@XmlElement
public void setAnyElement(final Element anyElement) {
this.anyElement = anyElement;
}
}
```

If I remove the modifier then Jackson behaves as expected. I can see the annotation introspector is set in both cases appropriately but unsure as of yet why adding the modifier prevents the desired behavior.

It looks like it also rewrites the name correctly to the value in the XmlAttribute tag, it's just the transform to an attribute that is missing (from what I can tell).