using source-overlay errs on second restore

This issue shows that packrat can restore from bundles in a source overlay directory but is unable to use those installed packages on subsequent restore. We are seeing this problem while using the packrat cache, but the steps here show that the cache is not necessary.

• Create a toy package that we will "pretend" is on GitHub.
• Create a tar.gz bundle using R CMD build completelyPrivatePackage.
• Add that tar.gz into a "sources" directory with a fake-SHA filename.
• Create a toy project that uses this package.
• Restore that project (using the sources directory).
• Restore that project again.

PackratFormat: 1.4
PackratVersion: 0.4.4
RVersion: 3.2.0
Repos: CRAN=http://cran.rstudio.com

Package: completelyPrivatePackage
Source: github
Version: 0.1.0
Hash: 88e8036bdbc415864d0f021ce6017997
GithubRepo: completelyPrivatePackage
GithubUsername: this-user-does-not-exist
GithubRef: master
GithubSha1: decafbaddecafbaddecafbaddecafbaddecafbad
RemoteHost: api.github.com
RemoteRepo: completelyPrivatePackage
RemoteUsername: this-user-does-not-exist
RemoteRef: master
RemoteSha: decafbaddecafbaddecafbaddecafbaddecafbad

In my toy project, I have a source-overlay directory that contains:

source-overlay/completelyPrivatePackage/decafbaddecafbaddecafbaddecafbaddecafbad.tar.gz

Here is a restore script that uses this overlay directory:

sourceOverlay <- file.path(getwd(), "source-overlay")
Sys.setenv(R_PACKRAT_SRC_OVERLAY = sourceOverlay)
Sys.setenv(R_PACKRAT_SRC_DIR = file.path(tempdir(), "packrat-src"))

options(packrat.untrusted.packages = character())
options(
  packrat.verbose.cache = TRUE,
  packrat.connect.timeout = 10
)

packrat::set_opts(
  auto.snapshot = FALSE,
  project = getwd(),
  persist = FALSE
)

packrat::restore(
  overwrite.dirty = TRUE,
  prompt = FALSE,
  restart = FALSE
)

On first run (no packrat/lib):

R --vanilla -s -f ./doit.R
Installing completelyPrivatePackage (0.1.0) ...
	OK (built source)
Warning messages:
1: In packrat::restore(overwrite.dirty = TRUE, prompt = FALSE, restart = FALSE) :
  The most recent snapshot was generated using R version 3.2.0
2: In untar(src, exdir = target, compressed = "gzip") :
  argument 'compressed' is ignored for the internal method
3: In untar(src, compressed = "gzip", list = TRUE) :
  argument 'compressed' is ignored for the internal method

On second run (after that successful restore):

R --vanilla -s -f ./doit.R
Error: Unable to retrieve package records for the following packages:
- 'completelyPrivatePackage'
In addition: Warning message:
In packrat::restore(overwrite.dirty = TRUE, prompt = FALSE, restart = FALSE) :
  The most recent snapshot was generated using R version 3.2.0
Execution halted

The restore works by adding the following fields to the DESCRIPTION file within that sources tar.gz file:

Source: github
GithubRepo: completelyPrivatePackage
GithubUsername: this-user-does-not-exist
GithubRef: master
GithubSHA1: decafbaddecafbaddecafbaddecafbaddecafbad
RemoteHost: api.github.com
RemoteRepo: completelyPrivatePackage
RemoteUsername: this-user-does-not-exist
RemoteRef: master
RemoteSha: decafbaddecafbaddecafbaddecafbaddecafbad

With this additional metadata, the second restore succeeds:

R --vanilla -s -f ./doit.R
Already up to date.
Warning message:
In packrat::restore(overwrite.dirty = TRUE, prompt = FALSE, restart = FALSE) :
  The most recent snapshot was generated using R version 3.2.0

Note: You'll need to remove the packrat/lib when switching between the original and augmented source bundles.

It feels as if packrat is seeing the presence of a tar.gz in the sources directory:
https://github.com/rstudio/packrat/blob/448aafd176ec6cbde307f556c91b39e5a1c94f9d/R/restore.R#L91-L92
That prevents us from adding DESCRIPTION fields associated with GitHub packages:
https://github.com/rstudio/packrat/blob/448aafd176ec6cbde307f556c91b39e5a1c94f9d/R/restore.R#L307-L328

We add fields for other types of repositories (bitbucket, gitlab, etc). Packages from these sources probably have the same issue.

Start with the supplied restore script and reproduce the first and second restores, then read R/restore.R around lines 91-92 and 307-328. Trace how the source-overlay tarball is handled and how repository metadata is added; done means a second restore succeeds without manually adding GitHub fields to DESCRIPTION.