Long-lived cookies might have unintended consequences on a crawling session

In a [recent investigation](http://ws-dl.blogspot.com/2018/03/2018-03-21-cookies-are-why-your.html) we found that about half of the Twitter pages are archived in non-English languages. Also, about half of those non-English captures are in [Kannada language](https://en.wikipedia.org/wiki/Kannada) alone. The root cause of this unintended consequence was found to be a sticky cookie.

Here is why it happens. Twitter pages have 47 alternate language links like this:

```

...

```

These links are added in the frontier queue every once in a while. Once any of these links are fetched, Twitter sets an explicit long-lasting `lang` cookie with the corresponding language. After that, every Twitter page is served in that sticky language unless it includes an explicit `?lang=` query parameter in the URL. In this list of alternate links, Kannada (`kn`) happens to be the last one, which overwrites any previous language cookies and affects all upcoming twitter links in that crawling session.

This is not a Twitter-specific issue. Similar consequences might exist in other places. A potential solution to this issue is to never let any cookie live for too long, instead, explicitly expire or remove them after a short period of time.

Read more details about our investigation of the matter in our blog post http://ws-dl.blogspot.com/2018/03/2018-03-21-cookies-are-why-your.html.