Should ListField support Nones for non-required sub-fields?

I haven't fully thought this through and I'm just logging an issue @JohnAD brought up in https://github.com/MongoEngine/mongoengine/issues/1106#issuecomment-269027633. We should think through all the use cases and discuss what's the best approach.

Right now, you can store None in any non-required field, e.g.

In [46]: class EmbeddedDoc(EmbeddedDocument):
    text = StringField()
   ....:

In [47]: class Doc(Document):
    name = StringField()
    num = IntField()
    embedded = EmbeddedDocumentField(EmbeddedDoc)
   ....:

In [48]: Doc.objects.create(name=None, num=None, embedded=None)
Out[48]: <Doc: Doc object>

Of course, that capability goes away once you define a field as required:

In [50]: Doc.objects.create(name=None, num=None, embedded=None)
---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
<ipython-input-50-eb46633914b2> in <module>()
----> 1 Doc.objects.create(name=None, num=None, embedded=None)

/Users/wojcikstefan/Repos/temp/mongoengine/mongoengine/queryset/base.pyc in create(self, **kwargs)
    286         .. versionadded:: 0.4
    287         """
--> 288         return self._document(**kwargs).save()
    289
    290     def first(self):

/Users/wojcikstefan/Repos/temp/mongoengine/mongoengine/document.pyc in save(self, force_insert, validate, clean, write_concern, cascade, cascade_kwargs, _refs, save_condition, signal_kwargs, **kwargs)
    318
    319         if validate:
--> 320             self.validate(clean=clean)
    321
    322         if write_concern is None:

/Users/wojcikstefan/Repos/temp/mongoengine/mongoengine/base/document.py in validate(self, clean)
    400                 pk = self._instance.pk
    401             message = 'ValidationError (%s:%s) ' % (self._class_name, pk)
--> 402             raise ValidationError(message, errors=errors)
    403
    404     def to_json(self, *args, **kwargs):

ValidationError: ValidationError (Doc:None) (Field is required: ['num', 'name', 'embedded'])

Now, I can imagine that we'd expect the same behavior from particular items within the ListField, namely that lists of non-required strings/ints/embedded docs would allow Nones in the list. That is not the case right now:

In [55]: class Doc(Document):
    names = ListField(StringField())
    nums = ListField(IntField())
    embedded_docs = ListField(EmbeddedDocumentField(EmbeddedDoc))
   ....:

In [56]: Doc.objects.create(names=[None], nums=[1], embedded_docs=[EmbeddedDoc(text='aaa')])
---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
<ipython-input-56-c349daa935c2> in <module>()
----> 1 Doc.objects.create(names=[None], nums=[1], embedded_docs=[EmbeddedDoc(text='aaa')])

/Users/wojcikstefan/Repos/temp/mongoengine/mongoengine/queryset/base.pyc in create(self, **kwargs)
    286         .. versionadded:: 0.4
    287         """
--> 288         return self._document(**kwargs).save()
    289
    290     def first(self):

/Users/wojcikstefan/Repos/temp/mongoengine/mongoengine/document.pyc in save(self, force_insert, validate, clean, write_concern, cascade, cascade_kwargs, _refs, save_condition, signal_kwargs, **kwargs)
    318
    319         if validate:
--> 320             self.validate(clean=clean)
    321
    322         if write_concern is None:

/Users/wojcikstefan/Repos/temp/mongoengine/mongoengine/base/document.py in validate(self, clean)
    400                 pk = self._instance.pk
    401             message = 'ValidationError (%s:%s) ' % (self._class_name, pk)
--> 402             raise ValidationError(message, errors=errors)
    403
    404     def to_json(self, *args, **kwargs):

ValidationError: ValidationError (Doc:None) (StringField only accepts string values: ['names'])

In [57]: Doc.objects.create(names=['name'], nums=[None], embedded_docs=[EmbeddedDoc(text='aaa')])
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-57-d6ac608e67d9> in <module>()
----> 1 Doc.objects.create(names=['name'], nums=[None], embedded_docs=[EmbeddedDoc(text='aaa')])

/Users/wojcikstefan/Repos/temp/mongoengine/mongoengine/queryset/base.pyc in create(self, **kwargs)
    286         .. versionadded:: 0.4
    287         """
--> 288         return self._document(**kwargs).save()
    289
    290     def first(self):

/Users/wojcikstefan/Repos/temp/mongoengine/mongoengine/base/document.py in __init__(self, *args, **values)
    113                         field = self._fields.get(key)
    114                         if field and not isinstance(field, FileField):
--> 115                             value = field.to_python(value)
    116                     setattr(self, key, value)
    117                 else:

/Users/wojcikstefan/Repos/temp/mongoengine/mongoengine/base/fields.pyc in to_python(self, value)
    304             self.field._auto_dereference = self._auto_dereference
    305             value_dict = {key: self.field.to_python(item)
--> 306                           for key, item in value.items()}
    307         else:
    308             Document = _import_class('Document')

/Users/wojcikstefan/Repos/temp/mongoengine/mongoengine/base/fields.pyc in <dictcomp>((key, item))
    304             self.field._auto_dereference = self._auto_dereference
    305             value_dict = {key: self.field.to_python(item)
--> 306                           for key, item in value.items()}
    307         else:
    308             Document = _import_class('Document')

/Users/wojcikstefan/Repos/temp/mongoengine/mongoengine/fields.py in to_python(self, value)
    179     def to_python(self, value):
    180         try:
--> 181             value = int(value)
    182         except ValueError:
    183             pass

TypeError: int() argument must be a string or a number, not 'NoneType'

In [58]: Doc.objects.create(names=['name'], nums=[1], embedded_docs=[None])
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-58-e79991fbb4fe> in <module>()
----> 1 Doc.objects.create(names=['name'], nums=[1], embedded_docs=[None])

/Users/wojcikstefan/Repos/temp/mongoengine/mongoengine/queryset/base.pyc in create(self, **kwargs)
    286         .. versionadded:: 0.4
    287         """
--> 288         return self._document(**kwargs).save()
    289
    290     def first(self):

/Users/wojcikstefan/Repos/temp/mongoengine/mongoengine/base/document.py in __init__(self, *args, **values)
    113                         field = self._fields.get(key)
    114                         if field and not isinstance(field, FileField):
--> 115                             value = field.to_python(value)
    116                     setattr(self, key, value)
    117                 else:

/Users/wojcikstefan/Repos/temp/mongoengine/mongoengine/base/fields.pyc in to_python(self, value)
    304             self.field._auto_dereference = self._auto_dereference
    305             value_dict = {key: self.field.to_python(item)
--> 306                           for key, item in value.items()}
    307         else:
    308             Document = _import_class('Document')

/Users/wojcikstefan/Repos/temp/mongoengine/mongoengine/base/fields.pyc in <dictcomp>((key, item))
    304             self.field._auto_dereference = self._auto_dereference
    305             value_dict = {key: self.field.to_python(item)
--> 306                           for key, item in value.items()}
    307         else:
    308             Document = _import_class('Document')

/Users/wojcikstefan/Repos/temp/mongoengine/mongoengine/fields.py in to_python(self, value)
    544     def to_python(self, value):
    545         if not isinstance(value, self.document_type):
--> 546             return self.document_type._from_son(value, _auto_dereference=self._auto_dereference)
    547         return value
    548

/Users/wojcikstefan/Repos/temp/mongoengine/mongoengine/base/document.py in _from_son(cls, son, _auto_dereference, only_fields, created)
    681         # Get the class name from the document, falling back to the given
    682         # class if unavailable
--> 683         class_name = son.get('_cls', cls._class_name)
    684
    685         # Convert SON to a dict, making sure each key is a string

AttributeError: 'NoneType' object has no attribute 'get'

As you can see, embedded document is completely broken (trying to call _from_son w/ None), but even the non-required strings and ints don't validate None properly...

Whatever we do here, we should consider performance implications, backward compatibility, etc. before making the final decision.

Start by reading mongoengine/base/fields.py and mongoengine/fields.py, especially ListField and the field to_python paths shown in the traceback; compare them with validation in mongoengine/base/document.py. Review the reported StringField, IntField, and EmbeddedDocumentField cases and the discussion linked from issue 1106. Done means an agreed, documented behavior for None in non-required list items, with compatibility and performance implications addressed.