tensorflow / tensorflow/datasets

downloading got Google Drive virus scan warning page rather than data files

Open
#3,935 7 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
4.6k
Forks
1.6k
Avg merge
3h 54m
Merged PRs (30d)
1

Description

/!\ PLEASE INCLUDE THE FULL STACKTRACE AND CODE SNIPPET

Short description
I am trying to load/download some datasets, mainly summarization ones. But instead of the *.tgz files, I got an HTML warning page telling me that the file is too large for Google Drive to scan virus and asking me whether to download anyway.

Environment information

  • Operating System: Ubuntu Linux 20.04

  • Python version: 3.10.4

  • tensorflow-datasets/tfds-nightly version: tensorflow-datasets 4.5.2

  • tensorflow/tf-nightly version: tensorflow 2.8.0

  • Does the issue still exists with the last tfds-nightly package (pip install --upgrade tfds-nightly) ?

N/A

Reproduction instructions

CoLab link here https://colab.research.google.com/drive/1F5jHy8o0_va6aIvuaB6H-EqfiWUrC9Ld#scrollTo=k3k-fYTuxw54

Python code below

import tensorflow_datasets as tfds
tfds.load('cnn_dailymail', split='test')

If you share a colab, make sure to update the permissions to share it.

Link to logs

I got the error message:

NonMatchingChecksumError: Artifact https://drive.google.com/uc?export=download&id=0BwmD_VLjROrfTHk4NFg2SndKcjQ, downloaded to /root/tensorflow_datasets/downloads/ucexport_download_id_0BwmD_VLjROrfTHk4NFg2SndKG8BdJPpt2iRo6Dpzz23CByJuAePEilB-pxbcBCHaWDs.tmp.cfa00e128d6c4efab209b7a281915239/uc, has wrong checksum. This might indicate:

Below is the downloaded file under my ~/tensorflow_datasets/download/<a random hash>:

<!DOCTYPE html><html><head><title>Google Drive - Virus scan warning</title><meta http-equiv="content-type" content="text/html; charset=utf-8"/><style nonce="sj0n0DT6hrEpyOgybFb8Iw">/* Copyright 2022 Google Inc. All Rights Reserved. */
.goog-inline-block{position:relative;display:-moz-inline-box;display:inline-block}* html .goog-inline-block{display:inline}*:first-child+html .goog-inline-block{display:inline}.goog-link-button{position:relative;color:#15c;text-decoration:underline;cursor:pointer}.goog-link-button-disabled{color:#ccc;text-decoration:none;cursor:default}body{color:#222;font:normal 13px/1.4 arial,sans-serif;margin:0}.grecaptcha-badge{visibility:hidden}.uc-main{padding-top:50px;text-align:center}#uc-dl-icon{display:inline-block;margin-top:16px;padding-right:1em;vertical-align:top}#uc-text{display:inline-block;max-width:68ex;text-align:left}.uc-error-caption,.uc-warning-caption{color:#222;font-size:16px}#uc-download-link{text-decoration:none}.uc-name-size a{color:#15c;text-decoration:none}.uc-name-size a:visited{color:#61c;text-decoration:none}.uc-name-size a:active{color:#d14836;text-decoration:none}.uc-footer{color:#777;font-size:11px;padding-bottom:5ex;padding-top:5ex;text-align:center}.uc-footer a{color:#15c}.uc-footer a:visited{color:#61c}.uc-footer a:active{color:#d14836}.uc-footer-divider{color:#ccc;width:100%}</style><link rel="icon" href="null"/></head><body><div class="uc-main"><div id="uc-dl-icon" class="image-container"><div class="drive-sprite-aux-download-file"></div></div><div id="uc-text"><p class="uc-warning-caption">Google Drive can't scan this file for viruses.</p><p class="uc-warning-subcaption"><span class="uc-name-size"><a href="/open?id=0BwmD_VLjROrfTHk4NFg2SndKcjQ">cnn_stories.tgz</a> (151M)</span> is too large for Google to scan for viruses. Would you still like to download this file?</p><form id="downloadForm" action="https://drive.google.com/uc?export=download&amp;id=0BwmD_VLjROrfTHk4NFg2SndKcjQ&amp;confirm=t" method="post"><input type="submit" id="uc-download-link" class="goog-inline-block jfk-button jfk-button-action" value="Download anyway"/></form></div></div><div class="uc-footer"><hr class="uc-footer-divider"></div></body></html>

Expected behavior
I was expecting to see progress bars of downloading for a while and then a return like this

 <PrefetchDataset element_spec={'abstract': TensorSpec(shape=(), dtype=tf.string, name=None), 'description': TensorSpec(shape=(), dtype=tf.string, name=None)}>

Additional context

This error repeats for many large datasets but not small ones. For example, I had no problem with cifar10. But I had the same issue with big_patent.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the reported Python snippet with the cnn_dailymail dataset and inspect the Google Drive download URL and resulting HTML file. Compare this behavior with the successful cifar10 download and the big_patent reproduction. Done means large dataset downloads yield the expected archive and pass checksum validation instead of returning the virus-scan warning page.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.