hcengineering / hcengineering/huly-selfhost

restore-workspace.sh reports success but all space contents are inaccessible after a full-instance restore

Open
#320 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
3.5k
Forks
469
PR merge metrics
No merged PRs in 30d

Description

### Summary

After a genuine disaster-recovery restore — volumes destroyed, stack rebuilt empty, admin
account recreated — `restore-workspace.sh` completes with exit code 0 and a green
`Done. Workspace '' restored.`, and **every object is written back to the database
correctly**. But the restored workspace is unusable: the UI and the REST API list the
spaces (projects, channels, funnels, drives, teamspaces) while showing **nothing inside
them**. No issues, no documents, no leads, no cards.

The data is intact. It is an access-control artefact — but to an operator it presents as
partial data loss, which is a bad failure mode for a restore path.

Reproduced twice, on two separate workspaces, on `v0.7.426`.

### Environment

- `huly-selfhost`, `HULY_VERSION=v0.7.426`, Docker Compose, single host (Ubuntu 24.04)
- CockroachDB / MinIO / Elasticsearch / Redpanda per the stock `compose.yml`
- Restore performed with the shipped `restore-workspace.sh`

### Steps to reproduce

```bash
# 1. Take a backup of a workspace that has real content inside its spaces
./backup-create.sh ./backups/myws myws

# 2. Destroy the instance completely — this is the case being tested
docker compose down -v
docker compose up -d
# Verified empty: global_account.workspace and global_account.social_id both return 0 rows.

# 3. Restore
./restore-workspace.sh ./backups/myws myws -e admin@example.com --no-merge -y

# 4. Wait for the workspace migration to finish, then log in and look
```

### Expected

The restoring admin can see the restored content.

### Actual

Exit code 0. `Done. Workspace 'myws' restored.` Spaces are listed; their contents are not
visible to the account that performed the restore.

Counted over `POST /_transactor/api/v1/find-all/{ws}` with a workspace-scoped token, and
confirmed on screen:

| class | after restore | actually in the database |
| --- | --- | --- |
| `tracker:class:Project` | 2 | 2 |
| `tracker:class:Issue` | **0** | **4** |
| `tracker:class:Milestone` | **0** | **2** |
| `tracker:class:Component` | **0** | **1** |
| `document:class:Document` | **0** | **1** |
| `lead:class:Lead` | **0** | **1** |
| `card:class:Card` | **0** | **3** |
| `chunter:class:Channel` | 2 | 2 |
| `drive:class:Drive` | 2 | 2 |

Pattern: objects **in** the `space` domain are visible; objects **scoped to** a space are
not.

This is not the known post-restore migration delay — the counts were unchanged 15+ minutes
later, and restarting `transactor` and `fulltext` made no difference.

### What is actually happening

During the restore, the `account.socialId` domain fails, five times:

```
failed to process domain
err={"message":"insert on table \"social_id\" violates foreign key constraint
\"social_id_person_fk\"", ...,
"detail":"Key (person_uuid)=('ff6cb7e2-…') is not present in table \"person\"",
"constraint_name":"social_id_person_fk", "code":"23503"}
domain="account.socialId"
```

The backup directory contains `account.socialId` but **no `account.person`**:

```
$ ls backups/myws/000001/ | sed 's/-[0-9]\{10,\}-[0-9]*\..*//' | sort -u
_migrations account.socialId activity blob calendar card channel chunter
collaborator contact document documents event hr love model_tx notification-dnc
preference sequence space tags task time tracker tx
```

So the original `person` row cannot be recreated on a fresh instance, and the social
identity restore can never satisfy its foreign key. The restore logs it and continues.

Meanwhile `restore-workspace.sh` created a **new** account with a **new** person UUID, and
`--no-merge` then restored the `space` rows verbatim — still pointing at the destroyed one:

```sql
SELECT _class, data->>'name', members, data->'owners' FROM public.space WHERE "workspaceId" = '…';

tracker:class:Project | Presens | {ff6cb7e2-…} | ["ff6cb7e2-…"] <- destroyed person
document:class:Teamspace | docs | {ff6cb7e2-…} | ["ff6cb7e2-…"]
lead:class:Funnel | Onboarding | {ff6cb7e2-…} | ["ff6cb7e2-…"]
```

The new account is `OWNER` in `global_account.workspace_members`, but it is a member of no
space — so space-level security filters out every space-scoped object.

Ordering matters here: `restore-workspace.sh` runs `set-user-role OWNER` at step 4 and the
restore at step 5, so `--no-merge` overwrites the membership that was just granted.

### Workaround

Re-point the orphaned space ACLs at the account that now exists, then restart the
transactor:

```sql
UPDATE public.space
SET members = array_replace(members, '', '')
WHERE "workspaceId" = '' AND '' = ANY(members);

UPDATE public.space
SET members = ARRAY['']
WHERE "workspaceId" = ''
AND array_length(members, 1) IS NULL
AND data->'owners' @> '[""]'::jsonb;

UPDATE public.space
SET data = jsonb_set(data, '{owners}', '[""]'::jsonb)
WHERE "workspaceId" = '' AND data->'owners' @> '[""]'::jsonb;
```

Careful: the system account owns the `core:class:SystemSpace` rows and must be left alone.

After this plus `docker compose restart transactor`, all 17 classes matched the pre-backup
counts exactly, on both workspaces tested.

### Suggested fixes, roughly in order of preference

1. **Have `restore-workspace.sh` re-point space ACLs after the restore**, since it already
knows both the workspace and the account it just created. That closes the gap without
changing the backup format.
2. **Fail loudly on `account.socialId` errors** rather than logging and continuing with
exit code 0. A restore that could not restore identities is not a successful restore.
3. **Include `account.person` in the backup**, so the original identity is reconstructible.
4. **Extend the verification script from platform#10999** (`compareDomainDigest`) to check
*reachability*, not only presence. Digest comparison would have passed here — every
document was restored. The failure is only visible when you query as a real user.

### Why this is worth fixing

The whole point of this path is the case where the machine is gone. In that case the
operator always has a new account, so this is not an edge case — it is the default
outcome of the disaster it exists for. And the symptom (projects listed, issues missing)
looks far more like data loss than like a permissions problem, which is the worst way for
it to present at the moment someone is already having a bad day.

Happy to test a patch — the reproduction is scripted on this end.

Contributor guide

No contributing guide indexed for this repository

Research direction

Read restore-workspace.sh around step 4, set-user-role OWNER, and step 5, the restore, then trace how account.socialId failures are handled. Reproduce the full-instance restore and inspect space members, owners, and visibility as the restored account; compare the reachability checks with platform#10999's compareDomainDigest. Done means a fresh restore exposes the restored space contents and reports identity-restore failures appropriately.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose, shell, sql
Domain
databases, devops, infrastructure, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.