hcengineering / hcengineering/huly-selfhost
restore-workspace.sh reports success but all space contents are inaccessible after a full-instance restore
- Dominant language
- Shell
- Stars
- 3.5k
- Forks
- 469
- PR merge metrics
- No merged PRs in 30d
Description
### Summary
After a genuine disaster-recovery restore — volumes destroyed, stack rebuilt empty, admin
account recreated — `restore-workspace.sh` completes with exit code 0 and a green
`Done. Workspace '' restored.`, and **every object is written back to the database
correctly**. But the restored workspace is unusable: the UI and the REST API list the
spaces (projects, channels, funnels, drives, teamspaces) while showing **nothing inside
them**. No issues, no documents, no leads, no cards.
The data is intact. It is an access-control artefact — but to an operator it presents as
partial data loss, which is a bad failure mode for a restore path.
Reproduced twice, on two separate workspaces, on `v0.7.426`.
### Environment
- `huly-selfhost`, `HULY_VERSION=v0.7.426`, Docker Compose, single host (Ubuntu 24.04)
- CockroachDB / MinIO / Elasticsearch / Redpanda per the stock `compose.yml`
- Restore performed with the shipped `restore-workspace.sh`
### Steps to reproduce
```bash
# 1. Take a backup of a workspace that has real content inside its spaces
./backup-create.sh ./backups/myws myws
# 2. Destroy the instance completely — this is the case being tested
docker compose down -v
docker compose up -d
# Verified empty: global_account.workspace and global_account.social_id both return 0 rows.
# 3. Restore
./restore-workspace.sh ./backups/myws myws -e admin@example.com --no-merge -y
# 4. Wait for the workspace migration to finish, then log in and look
```
### Expected
The restoring admin can see the restored content.
### Actual
Exit code 0. `Done. Workspace 'myws' restored.` Spaces are listed; their contents are not
visible to the account that performed the restore.
Counted over `POST /_transactor/api/v1/find-all/{ws}` with a workspace-scoped token, and
confirmed on screen:
| class | after restore | actually in the database |
| --- | --- | --- |
| `tracker:class:Project` | 2 | 2 |
| `tracker:class:Issue` | **0** | **4** |
| `tracker:class:Milestone` | **0** | **2** |
| `tracker:class:Component` | **0** | **1** |
| `document:class:Document` | **0** | **1** |
| `lead:class:Lead` | **0** | **1** |
| `card:class:Card` | **0** | **3** |
| `chunter:class:Channel` | 2 | 2 |
| `drive:class:Drive` | 2 | 2 |
Pattern: objects **in** the `space` domain are visible; objects **scoped to** a space are
not.
This is not the known post-restore migration delay — the counts were unchanged 15+ minutes
later, and restarting `transactor` and `fulltext` made no difference.
### What is actually happening
During the restore, the `account.socialId` domain fails, five times:
```
failed to process domain
err={"message":"insert on table \"social_id\" violates foreign key constraint
\"social_id_person_fk\"", ...,
"detail":"Key (person_uuid)=('ff6cb7e2-…') is not present in table \"person\"",
"constraint_name":"social_id_person_fk", "code":"23503"}
domain="account.socialId"
```
The backup directory contains `account.socialId` but **no `account.person`**:
```
$ ls backups/myws/000001/ | sed 's/-[0-9]\{10,\}-[0-9]*\..*//' | sort -u
_migrations account.socialId activity blob calendar card channel chunter
collaborator contact document documents event hr love model_tx notification-dnc
preference sequence space tags task time tracker tx
```
So the original `person` row cannot be recreated on a fresh instance, and the social
identity restore can never satisfy its foreign key. The restore logs it and continues.
Meanwhile `restore-workspace.sh` created a **new** account with a **new** person UUID, and
`--no-merge` then restored the `space` rows verbatim — still pointing at the destroyed one:
```sql
SELECT _class, data->>'name', members, data->'owners' FROM public.space WHERE "workspaceId" = '…';
tracker:class:Project | Presens | {ff6cb7e2-…} | ["ff6cb7e2-…"] <- destroyed person
document:class:Teamspace | docs | {ff6cb7e2-…} | ["ff6cb7e2-…"]
lead:class:Funnel | Onboarding | {ff6cb7e2-…} | ["ff6cb7e2-…"]
```
The new account is `OWNER` in `global_account.workspace_members`, but it is a member of no
space — so space-level security filters out every space-scoped object.
Ordering matters here: `restore-workspace.sh` runs `set-user-role OWNER` at step 4 and the
restore at step 5, so `--no-merge` overwrites the membership that was just granted.
### Workaround
Re-point the orphaned space ACLs at the account that now exists, then restart the
transactor:
```sql
UPDATE public.space
SET members = array_replace(members, '', '')
WHERE "workspaceId" = '' AND '' = ANY(members);
UPDATE public.space
SET members = ARRAY['']
WHERE "workspaceId" = ''
AND array_length(members, 1) IS NULL
AND data->'owners' @> '[""]'::jsonb;
UPDATE public.space
SET data = jsonb_set(data, '{owners}', '[""]'::jsonb)
WHERE "workspaceId" = '' AND data->'owners' @> '[""]'::jsonb;
```
Careful: the system account owns the `core:class:SystemSpace` rows and must be left alone.
After this plus `docker compose restart transactor`, all 17 classes matched the pre-backup
counts exactly, on both workspaces tested.
### Suggested fixes, roughly in order of preference
1. **Have `restore-workspace.sh` re-point space ACLs after the restore**, since it already
knows both the workspace and the account it just created. That closes the gap without
changing the backup format.
2. **Fail loudly on `account.socialId` errors** rather than logging and continuing with
exit code 0. A restore that could not restore identities is not a successful restore.
3. **Include `account.person` in the backup**, so the original identity is reconstructible.
4. **Extend the verification script from platform#10999** (`compareDomainDigest`) to check
*reachability*, not only presence. Digest comparison would have passed here — every
document was restored. The failure is only visible when you query as a real user.
### Why this is worth fixing
The whole point of this path is the case where the machine is gone. In that case the
operator always has a new account, so this is not an edge case — it is the default
outcome of the disaster it exists for. And the symptom (projects listed, issues missing)
looks far more like data loss than like a permissions problem, which is the worst way for
it to present at the moment someone is already having a bad day.
Happy to test a patch — the reproduction is scripted on this end.
Contributor guide
No contributing guide indexed for this repository
Research direction
Read restore-workspace.sh around step 4, set-user-role OWNER, and step 5, the restore, then trace how account.socialId failures are handled. Reproduce the full-instance restore and inspect space members, owners, and visibility as the restored account; compare the reachability checks with platform#10999's compareDomainDigest. Done means a fresh restore exposes the restored space contents and reports identity-restore failures appropriately.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, docker-compose, shell, sql
- Domain
- databases, devops, infrastructure, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100