automl / automl/RobustDARTS

Question about Hessian Matrix Calculation

Open
#11 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
158
Forks
39
PR merge metrics
No merged PRs in 30d

Description

Hello, really appreciate your nice work.

I hope this message finds you well. I have two questions regarding the calculation of the Hessian matrix in your code. Specifically, I'm looking at the [function](https://github.com/automl/RobustDARTS/blob/3dec3fedaa1770614bbfd4b98f5299d946ea7ac6/src/search/analyze.py#L157) where you calculate the second-order derivatives for each parameter with respect to all parameters:
```Python
row = self.gradient(grad[j], inputs[i:], retain_graph=True)[j:]
```
(1) I wonder why only the [j:] part of the result is taken? Is it assumed that the derivative has no effect on the preceding parameters?

(2) Additionally, when assigning values, why is the assignment done as follows and could you please explain the reasoning behind these specific assignments?
```Python
out.data[ai, ai:].add_(row.clone().type_as(out).data) # ai's row
if ai + 1 < n:
out.data[ai + 1:, ai].add_(row.clone().type_as(out).data[1:]) # ai's column
```

Thank you very much for your time and effort in maintaining this project. Your help is greatly appreciated.

Best regards,
Shun Lu

Contributor guide

No contributing guide indexed for this repository

Research direction

Read src/search/analyze.py at the linked line and trace the gradient calls that produce row and out. Determine how the j: slice and the row/column assignments represent the Hessian calculation, then document the reasoning in the issue or nearby documentation. No test or other entry point is mentioned.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.