Getting the deb up to Debian standards

I attach here the research that @ThomasGoirand did to get it up to par:

> I'm an experienced Debian Developer. I'm used to maintain big Python
> suites, and I'm not really scared by the amount of work needed to
> package AW in Debian. FYI, I'm also the maintainer of OpenStack in
> Debian, for which I maintain about 400+ packages.
>
> (REDACTED PARAGRAPH)
>
> Being not happy with this [using a propietary time tracking tool], I started looking for alternatives, and saw
> ActivityWatch. It looks promising, and nice, so I wanted to get it into
> Debian proper.
>
> So I started packaging each and every module of ActivityWatch one by
> one, the Debian way. I made separate packages for:
> - activitywatch-client
> - activitywatch-core
> - activitywatch-qt
> - activitywatch-server
> - activitywatch-watcher-afk
> - activitywatch-watcher-window
>
> but also:
> - python-json-logger
> - python-more-itertools
> - python-strict-rfc3339
> - python-takethetime
>
> Since ActivityWatch fails in Debian Sid (due to a more recent Qt version
> which I believe AW doesn't support yet), I did a Buster backport version
> to get it a first try. Meaning that I had to backport these:
> - flask
> - jinja2
> - markupsafe
> - peewee
> - python3-aniso8601
> - python-attrs
> - python-flask-cors
> - python-flask-restplus
> - python-importlib-metadata
> - python-iso8601
> - python-itsdangerous
> - python-json-logger
> - python-jsonschema
> - python-more-itertools
> - python-pyrsistent
> - python-sphinxcontrib-log-cabinet
> - python-sphinx-issues
> - python-strict-rfc3339
> - python-takethetime
> - python-werkzeug
> - python-zipp
> - repo
>
> At the moment, I did some dirty hacks to make it work in Buster, with
> some backported Python modules. I believe I got up to the point that AW
> is actually logging my activity in a SQL database. When starting aw-qt,
> the icon doesn't display in the tray (I use mate...), but it seems
> active and alive still. I can right click on it, and interact with AW. I
> went to look into /home/zigo/.local/share/activitywatch/aw-server and
> saw that there's now entries (that look proper) in the SQLite db.
> However, when I click on "Open Dashboard", firefox pops up, but then
> shows a 404, for a reason I don't understand yet.
>
> Now that you know what I've done so far, I have a few questions for you.
> Are you interested in having AW be in Debian proper (and then, as a
> consequence, reach Ubuntu universe) ? If yes, then I may help you by
> uploading what I've done so far. However, there's some work that needs
> to happen on your side, so that the application can work in Debian.
>
> This means first, get it to work with the current version of Qt in
> Debian Sid. This also means helping me fixing the 404 and find out
> what's going on, and repair.
>
> This also means that you must stop having requirements where you're
> pinning a specific version of things. It's ok to do >=, but it's really
> not if you do ===. Because in an environment like Debian, you do not
> control the version of Python modules: you just get whatever is the
> latest version uploaded by someone. If you need a higher version of a
> Python module, that's normally not a problem we can make it happen. But
> you must follow what's going on, and adapt whenever someone pushes
> something newer. In general, the latest version is always the right one.
> Would you be ok to work this way?
>
> Are you also ok to commit for *very* long period of time? If we upload
> AW in Debian now, this means we commit ourselves for the next 5 to 10
> years fixing issues. Otherwise, it's not worse it. I've seen that AW has
> been around for quite some time already, so you're probably ok with
> that, but I need to be sure I'll get your support if I commit myself
> into maintaining these packages in Debian.
>
> One question also: will there be a way so that I can publish my activity
> log in a remote server, once I have all up and running, and grant the
> access to my boss? Will it scale up to 100 employee?
>
> I guess that's it for the moment, I'll wait for your reply, and we'll
> see how we can proceed further if I get some kind of positive answer
> from you.

My reply:

> First off, thanks a lot for doing this research and getting as far as you have!
>
> >Are you interested in having AW be in Debian proper (and then, as a consequence, reach Ubuntu universe)?
>
> We're definitely interested in getting deb packages built, but I don't think we're willing to do specific LTS releases (if I understand you correctly), because we don't want to be slowed down by having to maintain old releases.
>
> >This also means that you must stop having requirements where you're pinning a specific version of things.
>
> The main reason we're pinning dependencies is because of PyInstaller which bundles them all up (to avoid having to depend on system packages) and sometimes chokes on newer versions (like PyQt). If you build a proper deb package you wouldn't use PyInstaller and you should, therefore, be able to use all the latest versions of the dependencies without issue.
>
> >One question also: will there be a way so that I can publish my activity
> >log in a remote server, once I have all up and running, and grant the
> >access to my boss? Will it scale up to 100 employee?
>
> This has been answered in part on the forum a few times. The short answer is: It can, but it isn't built with the use-case explicitly in mind so it's not a very smooth experience (yet). However, we're always *very* welcoming of companies willing to sponsor development to help advance that particular use-case.
>
> I'd love to answer the rest of your questions in more detail, but before I do, could you make an issue on GitHub or on the forum so the discussion is public? It's fine to just copy the email. Thanks :)

@thomasgoirand reply:

> Maintaining in Stable needs mainly to just backport security fixes,
> nothing more. It's usually not a lot of work, but when there's something
> to fix, it's nice if the backport can be done in time.
>
> >>This also means that you must stop having requirements where you're
> > pinning a specific version of things.
> >
> > The main reason we're pinning dependencies is because of PyInstaller
> > which bundles them all up (to avoid having to depend on system packages)
> > and sometimes chokes on newer versions (like PyQt). If you build a
> > proper deb package you wouldn't use PyInstaller and you should,
> > therefore, be able to use all the latest versions of the dependencies
> > without issue.
>
> Though the problem is that when starting-up, aw-qt checks for its
> requirements, and fails, unless I patch the requirements.txt file (which
> I did).
>
> >>One question also: will there be a way so that I can publish my activity
> >>log in a remote server, once I have all up and running, and grant the
> >>access to my boss? Will it scale up to 100 employee?
> >
> > This has been answered in part on the forum a few times. The short
> > answer is: It can, but it isn't built with the use-case explicitly in
> > mind so it's not a very smooth experience (yet).
>
> We're a hosting company, and most of us are engineers, so if there's a
> simple howto, maybe we can figure it out (I could script it, etc.).
>
> > However, we're always
> > *very* welcoming of companies willing to sponsor development to help
> > advance that particular use-case.
>
> I can talk about it with my boss, but I must first show him some
> results, which means, having the server part working (ie: no 404). I'll
> ask him how much money we're spending on Desktime, maybe we could
> redirect this amount onto your project rather than funding proprietary code.
>
> > I'd love to answer the rest of your questions in more detail, but before
> > I do, could you make an issue on GitHub or on the forum so the
> > discussion is public? It's fine to just copy the email. Thanks :)
>
> It is my opinion that it is too early to make it public. Very much, we
> can once we have something that works.
>
> Now, how can we move forward and solve the issue with aw-server only
> showing some 404? Can I just give you the bunch of packages that needs
> to be installed in Buster, or maybe a virtualbox image, so you can try
> to debug it?

My reply:

> The 404 is probably because you haven't built the web UI and put the files in the right folder. The Makefile in aw-server should show how that is done.
>
> If you don't mind me asking, how come you really don't want to use the pre-built zips? (I can imagine many reasons, but I want to hear yours)
>
> I don't really believe in "public when ready", our entire development process is open and I'm not the only ActivityWatch developer interested in this. It would be easier and you'd get answers much quicker if we could continue this in public.
>
> I will have to think about the rest of the stuff later next week, trying not to work all weekend.

@thomasgoirand reply:

>
> I don't think I made myself clear, so let me try again! :)
>
> In Debian, the bulk of the work is done in Sid. It's not hard to
> maintain for Stable. It is a way harder to keep the package always
> working in Debian unstable, where things are moving fast. Your package
> must be able to accept any new version of any module that it depends on.
> As the freeze for the next stable approaches, it becomes even more
> important to react fast.
>
> Then after, in Stable, if there's no bugs in the version there, and no
> security patches, it's zero work, most of the time...
>
> On 3/28/20 9:17 PM, Erik Bjäreholt wrote:
> > The 404 is probably because you haven't built the web UI and put the
> > files in the right folder. The Makefile in aw-server should show how
> > that is done.
>
> I don't believe I did that indeed. I went to look into the Makefile, but
> didn't understand how it's doing things. The make file does:
>
> make --directory=aw-webui build
>
> though for me, the result is:
>
> make: Entering directory
> '/home/zigo/salsa/debian/activitywatch/activitywatch-server/activitywatch-server/aw-webui'
> make: *** No rule to make target 'build'. Stop.
> make: Leaving directory
> '/home/zigo/salsa/debian/activitywatch/activitywatch-server/activitywatch-server/aw-webui'
>
> How is this suppose to work?
>
> Note that I am downloading each individual project separately, and
> didn't use the git submodule thing (which IMO, it'd be best you gave up
> on: git submodule is a horrible concept). That's probably why. I just
> saw this:
>
> https://github.com/ActivityWatch/aw-webui
>
> It looks like using npm. Will it download so many JS file?!? If so, then
> that's probably a lot of work: I'll need to get all of the nodejs /
> libjs files packaged individually. Hopefully, some are already in
> Debian, but then that's the same problem as for Python: your program
> will need to be able to cope with whatever is in Debian.
>
> > If you don't mind me asking, how come you really don't want to use the
> > pre-built zips? (I can imagine many reasons, but I want to hear yours)
>
> There's many reasons, one of them is that I want your package to be in
> Debian, and the policy forbids using this type of packaging. We must
> also build from source, without downloading any content during build.
>
> Then there's the fact that I hate using venv which are an heresy for
> anyone doing packaging. Plus the fact that it wouldn't be using the
> libraries of the distributions, leading to potential security problems,
> especially considering that you're pinning everything to a specific version.
>
> Finally, it feels like doing "curl foo | bash". I don't do that on my
> laptop, and certainly, that's not what I want to provide to my
> colleagues as a solution.
>
>On 3/28/20 9:38 PM, Thomas Goirand wrote:
> > I just saw this:
> >
> > https://github.com/ActivityWatch/aw-webui
> >
> > It looks like using npm. Will it download so many JS file?!? If so, then
> > that's probably a lot of work: I'll need to get all of the nodejs /
> > libjs files packaged individually. Hopefully, some are already in
> > Debian [...]
> I ran "npm install" in a freshly checked-out aw-webui Git repo, then
> compared the resulting node_modules folder content with what's in the
> Debian archive (it's easy and fast, as I have a Debian mirror on my
> laptop). This generated a list of 160 missing nodejs modules that would
> need to be packaged. At this point in time, I'm not sure I want to
> commit myself into doing that much work, unfortunately...
>
> I've attached the list of found and not JS dependencies in Debian.
>
> Is there a way to get some kind of browseable results with ActivityWatch
> without this JS storm?

@johan-bjareholt replied:

> The web-ui is just a frontend for the server, the only other frontend I'm aware of is workflow which has significantly less dependencies and is written in GTK which is nice.
> https://gitlab.com/cunidev/workflow
> The issues with it that I'm aware of is that it's lacking a lot of features, ignores the afk bucket so doesn't detect when the user is afk, not cross platform etc.
>
> I think there are multiple issues with packaging activitywatch to a distro right now. The web-ui and npm dependency hell is the major one, but the switch to aw-server-rust which we will switch to soon is another. There are probably more which we will discover if we continue working on it.
>
> The best solution if you want a .deb file soon would be to make our own PPA which I don't think we're interested in maintaining that. Maybe if we could get something like Open Build Service working for the project. This is necessary for being able to avoid packaging all npm dependencies for every distro so we would instead provide just a downloadable .deb/.rpm etc..
> Another option which I'm not sure if it's allowed in debian would be to let github make a "webui.zip" which the .deb build can download and extract when the package is being built. The web-ui is a single-page-application so the dependencies are not needed after they are built as they're all bundled together similar to static linking.
>
> I don't think official debian package is suitable for an application with so many npm dependencies and the web-ui is a component we have spent a lot of time on writing.
> If we ever get a proper GTK/Qt frontend it certainly would fit into debian, but not in its current state.

@thomasgoirand replied:

> On 3/29/20 6:02 PM, Johan Bjäreholt wrote:
> > The best solution if you want a .deb file soon would be to make our own
> > PPA which I don't think we're interested in maintaining that.
>
> I could write my own repository, and in fact, I was planning on doing
> this for the Buster backports, but that's not the long term way.
>
> > Maybe if
> > we could get something like Open Build Service working for the project.
> > This is necessary for being able to avoid packaging all npm dependencies
> > for every distro so we would instead provide just a downloadable
> > .deb/.rpm etc..
>
> Automating building of npm dependencies doesn't require something like
> OBS, so I wouldn't go this way. I'll have a look if I can see some
> automated ways to create some nodejs packages first.
>
> > Another option which I'm not sure if it's allowed in debian would be to
> > let github make a "webui.zip" which the .deb build can download and
> > extract when the package is being built.
>
> Downloading during build is forbidden. Using an opaque blob (like a .zip
> with all the .js files) is also forbidden. So that's not at all the way
> to go for Debian.
>
> > The web-ui is a
> > single-page-application so the dependencies are not needed after they
> > are built as they're all bundled together similar to static linking.
>
> Yes, and in Debian, we don't consider these as source code, and
> therefore, it cannot be in a source package.
>
> > I don't think official debian package is suitable for an application
> > with so many npm dependencies and the web-ui is a component we have
> > spent a lot of time on writing.
>
> I do. It just needs time and love. There's some major applications like
> Gitlab which are currently being worked on, which require hundreds of
> nodejs packages. It's doable.
>
> > If we ever get a proper GTK/Qt frontend it certainly would fit into
> > debian, but not in its current state.
> >
> > Mvh Johan
>
> What worries me more is being able to centralize the data in a single
> server, so my boss can look at what employees are doing (which is the
> final goal for me).
>
> As I understand, we just would need to sync the sqlite db into that
> central server, and have some kind of web UI to display it. Everything
> else from ActivityWatch is already there.
>
> I'll see with my boss what he thinks of paying you for doing that.

@johan-bjareholt replied:

> The issue I see with packaging a hundred NPM dependencies is that if any other Debian package would want another major or a higher minor version of some NPM package we would have to have that in mind when choosing package versions. The NPM ecosystem is young and has not made it easy to handle that easily as a lot of good packages have a lot of "micro dependencies". It's hard already as it is to handle our dependencies in aw-webui.
>
> I see it as much easier if we want this working quickly to just ship a pre-built webui (possibly even as an optional dependency to aw-server) and have that be an non-issue for now as I don't think that should be the priority if we want this use-case to work sooner rather than later.
>
> > As I understand, we just would need to sync the sqlite db into that
> central server, and have some kind of web UI to display it. Everything
> else from ActivityWatch is already there.
>
> Erik has been prototyping syncing, but only for personal use (a few devices at max).
>
> Our long term "enterprise" plan has been to send a summary of the activity (a report) to some central server. This would be much easier than to sync databases as each database in the long run can grow to a hundred megabytes per user and constantly changing. I think if we gave it a shot we can prototype such a solution somewhat quickly as we already have a pretty good tool set in both the web-ui and aw-server that we can re-use.

The last reply by @johan-bjareholt was on 29 march 2020. A of writing, there has been no new replies.