Lexical scope document control directives

Currently, document control directives such as :stopdoc:, :startdoc:, :enddoc:, :nodoc: are buggy.
Documentation and implementation are both wrong. Nobody knows the specification of these directives.
We need to make it clear what the expected specification is, what we are making and fixing.

My proposal is to introduce a lexical scope document control that follows class/module nesting.

# Preprocessor-style startdoc/stopdoc. But we don't write like this.
class A
  class B
# :startdoc:
    class C
      X = 1
# :stopdoc:
      Y = 2
    end
  end
end

Make :stopdoc: and :startdoc: lexical scope.

Don't modify container code object's state, just skip documenting in stopdoc region.
When reached an end that close a class/module, it also closes :startdoc: or :stopdoc: in the class/module scope.
This is similar to current behavior but clearly defines what happen if there is no corresponding :startdoc:.

class A
  # :stopdoc:
  class B
    # (stopdoc applied)
    # :startdoc:
    # (startdoc applied)
  end # (close startdoc at L5)
  # (stopdoc applied)
end # (close stopdoc at L2)

class A
  # :stopdoc:
  class B
    def nodoc; end
  end
  # :startdoc:
end

class A
  class B
    # not documented
    def shoulddoc; end
  end
end

Change :enddoc: to stop documenting in the current lexical scope

It will be almost the same as :stopdoc: but can't do startdoc.
Note that :enddoc: is rarely used (in ruby/*), mostly used at toplevel (in rails/rails).

class A
  class B
    # :enddoc:
    class C
      # (document disabled here too)
    end
  end # (enddoc ends here)
end

class B
  # (document enabled)
  # (originally, class B is marked as document_done)
end

class A
  class B
    def f; end
  end
  # :enddoc:
  class B # B is hidden from documentation
    def g; end
  end
end

class A::B
  def f; end
  # :enddoc:
  def g; end
end

class A
  class B
    # documented?
    def h; end
    class C # How about this?
    end
  end
end
class A::B::C # and this?
end

Make :doc: override :stopdoc: (optional)

This will expand use case of :doc: directive. It's just an enhancement idea. Not important.

class A
  private def f; end # :doc: (original usage)
  # :stopdoc:
  ...
  def f; end # :doc: (additional usage)
  ...
  # :startdoc:
end

Change :nodoc:all to globally mark code object as nodoc

Document for the same code object in any other files will be ignored.
Also applies something like :enddoc: to this scope. All method def including unrelated open class inside here is just ignored.
I think this will cover most usecase of :nodoc: to hide internal-use classes/modules.

# a.rb
class Internal # :nodoc: all
  class ::A; end # ignored
end

class A # (documented)
end

# b.rb
class Internal
  # (not documented)
end

Change :nodoc: to mark lexical scope as nodoc

Similar to :enddoc:. For compatibility, :nodoc: can't mark code object as hidden from document.

class Array # :nodoc:
  include CoreExt # (not documented)
  class B; end # (not documented)
  def f; end # (not documented)
end

class Array
  # (this scope is documented)
  def g; end
end

Actual cases that needs this compatibility:

class OptionParser
  # document comment for OptionParser::Switch
  class Switch
    # :nodoc: (Switch should be documented, only code objects inside should be hidden)

    attr_reader :pattern, :conv, :short, :long, :arg, :desc, :block
  end
end

module Net   #:nodoc: (Net should be documented. Only this scope should be ignored)
  # :stopdoc:
  class HTTPBadResponse < StandardError; end
  class HTTPHeaderSyntaxError < StandardError; end
  # :startdoc:
  ...
end

# mkmf.rb
class String # :nodoc:
  # Wraps a string in escaped quotes if it contains whitespace.
  def quote
    /\s/ =~ self ? "\"#{self}\"" : "#{self}"
  end
end

# pp.rb
class Array # :nodoc:
  def pretty_print(q) # :nodoc:
    q.group(1, '[', ']') { ... }
  end
end

class A
  class B
  end
  class B::C # :nodoc:
  end
end

class A::B # A::B is wrongly marked as nodoc
  def f; end
end

Make standalone intermediate :nodoc: and :nodoc: all no-op with warning

In the following intermediate :nodoc: usage, the range of :nodoc: is not obvious.
Someone may expect only after :nodoc: is nodoc, some may expect the whole class A ... end range is nodoc.
I think we should prohibit this usage

# Ambiguous prohibited :nodoc:
class A
  def f; end # Should we prevent this to be documented?

  # :nodoc: (prohibit this)

  # comment
  def g; end # This is not documented
end

# Accepted :nodoc:
class A
  # :nodoc: (Only at the beginning of class)
  # Nodoc directives for current class/module are only accepted
  # on the comment lines before any other comments or ruby codes

  #
  def f; end
  def g; end
end

Related to #1578

class A
  def f; end

  # method g is nodoc in #1578 proposal
  # :nodoc:
  def g; end

  # :nodoc: (this is prohibited in this proposal)

  # comment
  def h; end
end

Start by tracing RDoc's parsing phase and how document_self and document_children are handled for nested classes and modules. Compare the directive examples in this issue with the current behavior; done means the lexical-scope rules are specified, implemented consistently for all listed directives, and the existing bugs no longer reproduce.