py-why / py-why/causal-learn

ValueError: math domain error in PC with missing data

Open
#138 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.7k
Forks
274
PR merge metrics
No merged PRs in 30d

Description

Hi there,
my input data is like this:

image

I then want to discover with missing values:

from causallearn.search.ConstraintBased.PC import pc
dataset= X.to_numpy()
sub_cols = X.columns
# default parameters
cg = pc(dataset,alpha=0.05,indep_test='mv_fisherz',mvpc=True)

Full error:


ValueError                                Traceback (most recent call last)
Cell In[206], line 5
      3 sub_cols = X.columns
      4 # default parameters
----> 5 cg = pc(dataset,alpha=0.05,indep_test='mv_fisherz',mvpc=True)

File /opt/conda/lib/python3.10/site-packages/causallearn/search/ConstraintBased/PC.py:41, in pc(data, alpha, indep_test, stable, uc_rule, uc_priority, mvpc, correction_name, background_knowledge, verbose, show_progress, node_names, **kwargs)
     39     if indep_test == fisherz:
     40         indep_test = mv_fisherz
---> 41     return mvpc_alg(data=data, node_names=node_names, alpha=alpha, indep_test=indep_test, correction_name=correction_name, stable=stable,
     42                     uc_rule=uc_rule, uc_priority=uc_priority, background_knowledge=background_knowledge,
     43                     verbose=verbose,
     44                     show_progress=show_progress, **kwargs)
     45 else:
     46     return pc_alg(data=data, node_names=node_names, alpha=alpha, indep_test=indep_test, stable=stable, uc_rule=uc_rule,
     47                   uc_priority=uc_priority, background_knowledge=background_knowledge, verbose=verbose,
     48                   show_progress=show_progress, **kwargs)

File /opt/conda/lib/python3.10/site-packages/causallearn/search/ConstraintBased/PC.py:200, in mvpc_alg(data, node_names, alpha, indep_test, correction_name, stable, uc_rule, uc_priority, background_knowledge, verbose, show_progress, **kwargs)
    198 indep_test = CIT(data, indep_test, **kwargs)
    199 ## Step 1: detect the direct causes of missingness indicators
--> 200 prt_m = get_parent_missingness_pairs(data, alpha, indep_test, stable)
    201 # print('Finish detecting the parents of missingness indicators.  ')
    202 
    203 ## Step 2:
    204 ## a) Run PC algorithm with the 1st step skeleton;
    205 cg_pre = SkeletonDiscovery.skeleton_discovery(data, alpha, indep_test, stable,
    206                                               background_knowledge=background_knowledge,
    207                                               verbose=verbose, show_progress=show_progress, node_names=node_names)

File /opt/conda/lib/python3.10/site-packages/causallearn/search/ConstraintBased/PC.py:275, in get_parent_missingness_pairs(data, alpha, indep_test, stable)
    272 ## Get the index of parents of missingness indicators
    273 # If the missingness indicator has no parent, then it will not be collected in prt_m
    274 for missingness_i in missingness_index:
--> 275     parent_of_missingness_i = detect_parent(missingness_i, data, alpha, indep_test, stable)
    276     if not isempty(parent_of_missingness_i):
    277         parent_missingness_pairs['prt'].append(parent_of_missingness_i)

File /opt/conda/lib/python3.10/site-packages/causallearn/search/ConstraintBased/PC.py:363, in detect_parent(r, data_, alpha, indep_test, stable)
    361 if len(Neigh_x) >= depth:
    362     for S in combinations(Neigh_x, depth):
--> 363         p = cg.ci_test(x, y, S)
    364         if p > alpha:
    365             if not stable:  # Unstable: Remove x---y right away

File /opt/conda/lib/python3.10/site-packages/causallearn/graph/GraphClass.py:58, in CausalGraph.ci_test(self, i, j, S)
     56 # assert i != j and not i in S and not j in S
     57 if self.test.method == 'mc_fisherz': return self.test(i, j, S, self.nx_skel, self.prt_m)
---> 58 return self.test(i, j, S)

File /opt/conda/lib/python3.10/site-packages/causallearn/utils/cit.py:388, in MV_FisherZ.__call__(self, X, Y, condition_set)
    386 if abs(r) >= 1: r = (1. - np.finfo(float).eps) * np.sign(r) # may happen when samplesize is very small or relation is deterministic
    387 Z = 0.5 * log((1 + r) / (1 - r))
--> 388 X = sqrt(len(test_wise_deletion_XYcond_rows_index) - len(condition_set) - 3) * abs(Z)
    389 p = 2 * (1 - norm.cdf(abs(X)))
    390 self.pvalue_cache[cache_key] = p

ValueError: math domain error

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the call in search/ConstraintBased/PC.py with the reported missing-data configuration, then trace the failing calculation in utils/cit.py, especially MV_FisherZ.call. Check how few valid test-wise rows and conditioning-set size are handled, and add a regression test that defines the expected behavior instead of raising ValueError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.