alibaba / alibaba/FederatedScope
Issue with federate.method set to global
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 261
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
I have launched the experiment with command
` python federatedscope/main.py --cfg federatedscope/cv/baseline/fedavg_convnet2_on_cifar10.yaml federate.client_num 1 federate.sample_client_rate 1.0 federate.method global`
However, it looks the model is not updated during the evaluation time. The test accuracy stays at 11% for all the rounds while the training accuracy improves.
```
2024-04-29 16:42:48,817 (client:357) INFO: {'Role': 'Client #1', 'Round': 0, 'Results_raw': {'train_avg_loss': 1.446278, 'train_total': 50000, 'train_acc': 0.48616, 'train_correct': 24308.0, 'train_loss': 72313.922022}}
2024-04-29 16:42:48,820 (server:344) INFO: Server: Starting evaluation at the end of round 0.
2024-04-29 16:42:50,443 (context:296) WARNING: No val_data or val_loader in the trainer, will skip evaluation.If this is not the case you want, please check whether there is typo for the name
2024-04-29 16:42:50,445 (server:960) INFO: {'Role': 'Server #', 'Round': 1, 'Results_raw': {'test_avg_loss': 2.301025, 'test_total': 10000, 'test_acc': 0.11, 'test_correct': 1100.0, 'test_loss': 23010.249565}}
2024-04-29 16:42:50,445 (server:350) INFO: ----------- Starting a new training round (Round #1) -------------
2024-04-29 16:42:59,204 (client:357) INFO: {'Role': 'Client #1', 'Round': 1, 'Results_raw': {'train_avg_loss': 1.096129, 'train_total': 50000, 'train_acc': 0.6146, 'train_correct': 30730.0, 'train_loss': 54806.453947}}
2024-04-29 16:42:59,206 (server:344) INFO: Server: Starting evaluation at the end of round 1.
2024-04-29 16:43:00,697 (context:296) WARNING: No val_data or val_loader in the trainer, will skip evaluation.If this is not the case you want, please check whether there is typo for the name
2024-04-29 16:43:00,697 (server:960) INFO: {'Role': 'Server #', 'Round': 2, 'Results_raw': {'test_avg_loss': 2.301025, 'test_total': 10000, 'test_acc': 0.11, 'test_correct': 1100.0, 'test_loss': 23010.249565}}
2024-04-29 16:43:00,697 (server:350) INFO: ----------- Starting a new training round (Round #2) -------------
2024-04-29 16:43:09,473 (client:357) INFO: {'Role': 'Client #1', 'Round': 2, 'Results_raw': {'train_avg_loss': 0.959477, 'train_total': 50000, 'train_acc': 0.66432, 'train_correct': 33216.0, 'train_loss': 47973.853004}}
2024-04-29 16:43:09,474 (server:344) INFO: Server: Starting evaluation at the end of round 2.
2024-04-29 16:43:11,000 (context:296) WARNING: No val_data or val_loader in the trainer, will skip evaluation.If this is not the case you want, please check whether there is typo for the name
2024-04-29 16:43:11,000 (server:960) INFO: {'Role': 'Server #', 'Round': 3, 'Results_raw': {'test_avg_loss': 2.301025, 'test_total': 10000, 'test_acc': 0.11, 'test_correct': 1100.0, 'test_loss': 23010.249565}}
2024-04-29 16:43:11,000 (server:350) INFO: ----------- Starting a new training round (Round #3) -------------
2024-04-29 16:43:19,756 (client:357) INFO: {'Role': 'Client #1', 'Round': 3, 'Results_raw': {'train_avg_loss': 0.867314, 'train_total': 50000, 'train_acc': 0.6992, 'train_correct': 34960.0, 'train_loss': 43365.681585}}
2024-04-29 16:43:19,757 (server:344) INFO: Server: Starting evaluation at the end of round 3.
2024-04-29 16:43:21,245 (context:296) WARNING: No val_data or val_loader in the trainer, will skip evaluation.If this is not the case you want, please check whether there is typo for the name
2024-04-29 16:43:21,245 (server:960) INFO: {'Role': 'Server #', 'Round': 4, 'Results_raw': {'test_avg_loss': 2.301025, 'test_total': 10000, 'test_acc': 0.11, 'test_correct': 1100.0, 'test_loss': 23010.249565}}
2024-04-29 16:43:21,246 (server:350) INFO: ----------- Starting a new training round (Round #4) -------------
2024-04-29 16:43:29,954 (client:357) INFO: {'Role': 'Client #1', 'Round': 4, 'Results_raw': {'train_avg_loss': 0.794839, 'train_total': 50000, 'train_acc': 0.72466, 'train_correct': 36233.0, 'train_loss': 39741.947819}}
2024-04-29 16:43:29,956 (server:344) INFO: Server: Starting evaluation at the end of round 4.
2024-04-29 16:43:31,466 (context:296) WARNING: No val_data or val_loader in the trainer, will skip evaluation.If this is not the case you want, please check whether there is typo for the name
2024-04-29 16:43:31,466 (server:960) INFO: {'Role': 'Server #', 'Round': 5, 'Results_raw': {'test_avg_loss': 2.301025, 'test_total': 10000, 'test_acc': 0.11, 'test_correct': 1100.0, 'test_loss': 23010.249565}}
2024-04-29 16:43:31,467 (server:350) INFO: ----------- Starting a new training round (Round #5) -------------
2024-04-29 16:43:40,151 (client:357) INFO: {'Role': 'Client #1', 'Round': 5, 'Results_raw': {'train_avg_loss': 0.730278, 'train_total': 50000, 'train_acc': 0.74824, 'train_correct': 37412.0, 'train_loss': 36513.923683}}
2024-04-29 16:43:40,153 (server:344) INFO: Server: Starting evaluation at the end of round 5.
2024-04-29 16:43:41,618 (context:296) WARNING: No val_data or val_loader in the trainer, will skip evaluation.If this is not the case you want, please check whether there is typo for the name
2024-04-29 16:43:41,619 (server:960) INFO: {'Role': 'Server #', 'Round': 6, 'Results_raw': {'test_avg_loss': 2.301025, 'test_total': 10000, 'test_acc': 0.11, 'test_correct': 1100.0, 'test_loss': 23010.249565}}
2024-04-29 16:43:41,619 (server:350) INFO: ----------- Starting a new training round (Round #6) -------------
2024-04-29 16:43:50,265 (client:357) INFO: {'Role': 'Client #1', 'Round': 6, 'Results_raw': {'train_avg_loss': 0.671751, 'train_total': 50000, 'train_acc': 0.77108, 'train_correct': 38554.0, 'train_loss': 33587.532097}}
2024-04-29 16:43:50,266 (server:344) INFO: Server: Starting evaluation at the end of round 6.
2024-04-29 16:43:51,771 (context:296) WARNING: No val_data or val_loader in the trainer, will skip evaluation.If this is not the case you want, please check whether there is typo for the name
2024-04-29 16:43:51,771 (server:960) INFO: {'Role': 'Server #', 'Round': 7, 'Results_raw': {'test_avg_loss': 2.301025, 'test_total': 10000, 'test_acc': 0.11, 'test_correct': 1100.0, 'test_loss': 23010.249565}}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running the command in the issue with federatedscope/cv/baseline/fedavg_convnet2_on_cifar10.yaml and inspect federatedscope/main.py alongside the evaluation warning in the logs. Trace the global-method evaluation path and the missing val_data or val_loader condition; done means evaluation uses the updated model and test accuracy no longer remains fixed at 11%.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100