Postgres cartridge cannot change fingerprint type in bingo_idx
- Dominant language
- C++
- Stars
- 406
- Forks
- 134
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 24
Description
**Steps to Reproduce**
Environment:
- **Bingo cartridge (1.9.1.637-gafaea530d)**
- **Postgres 15.4**
1. Create index:
```sql
CREATE INDEX idx_t_molecule__canonical_smiless_bingo ON t_molecule USING bingo_idx (canonical_smiles bingo.molecule);
```
2. Got 100% similarity as long as the query mol is a substructure of result mol, which is not what we want.
```sql
-- tanimoto similarity
select canonical_smiles, bingo.getsimilarity(canonical_smiles, 'OB(O)c1ccccc1', 'tanimoto')
from t_molecule
where canonical_smiles @ (0.9, 1, 'OB(O)c1ccccc1', 'tanimoto')::bingo.sim
order by bingo.getsimilarity(canonical_smiles, 'OB(O)c1ccccc1', 'tanimoto') desc limit 10;
canonical_smiles | getsimilarity
--------------------------------------------+---------------
OB(O)c1ccccc1 | 1
OB(O)c1ccc(B(O)O)cc1 | 1 -- this one
[2H]OB(O[2H])c1ccccc1 | 1
[2H]c1ccccc1B(O)O | 1
[2H]c1c([2H])c([2H])c(B(O)O)c([2H])c1[2H] | 1
OB(O)[13c]1[13cH][13cH][13cH][13cH][13cH]1 | 1
OB(O)c1cc(B(O)O)cc(B(O)O)c1 | 0.95454544
OB(O)c1cccc(B(O)O)c1 | 0.95454544
OB(O)c1ccc2cc3ccccc3cc2c1 | 0.9130435
OB(O)c1cc2ccc3cccc4ccc(c1)c2c34 | 0.9130435
(10 rows)
Time: 721.495 ms
-- Cuz 'OB(O)c1ccccc1' & 'OB(O)c1ccc(B(O)O)cc1' has same 'sim' fingerprints,
-- which is the default fingerprint type: https://lifescience.opensource.epam.com/bingo/user-manual-oracle.html#molecule-fingerprints
select bingo.Fingerprint('OB(O)c1ccccc1', 'sim')=bingo.Fingerprint('OB(O)c1ccc(B(O)O)cc1', 'sim') as sim,
bingo.Fingerprint('OB(O)c1ccccc1', 'sub')=bingo.Fingerprint('OB(O)c1ccc(B(O)O)cc1', 'sub') as sub,
bingo.Fingerprint('OB(O)c1ccccc1', 'sub-res')=bingo.Fingerprint('OB(O)c1ccc(B(O)O)cc1', 'sub-res') as sub_res,
bingo.Fingerprint('OB(O)c1ccccc1', 'sub-tau')=bingo.Fingerprint('OB(O)c1ccc(B(O)O)cc1', 'sub-tau') as sub_tau,
bingo.Fingerprint('OB(O)c1ccccc1', 'full')=bingo.Fingerprint('OB(O)c1ccc(B(O)O)cc1', 'full') as full;
sim | sub | sub_res | sub_tau | full
-----+-----+---------+---------+------
t | f | f | f | f
```
3. Considering the size of `t_molecule` (tens of million) and the tanimoto similarity search speed, it's clearly Bingo cartridge is utilizing the fingerprint cache which was create during step 1 (as described in https://lifescience.opensource.epam.com/bingo/user-manual-oracle.html#creating-an-index). **To get a non-100% similary result on molecules like ('OB(O)c1ccccc1', 'OB(O)c1ccc(B(O)O)cc1')**, I have tried the following:
```sql
-- check if additional parameters can be used during creating index
-- https://github.com/epam/Indigo/blob/cc682584c5942742287637817c48eb4669813f8b/bingo/postgres/tests/local/postgres_test.sql#L9
CREATE INDEX idx_t_molecule__canonical_smiless_bingo ON t_molecule USING bingo_idx (canonical_smiles bingo.molecule) with (REJECT_INVALID_STRUCTURES=1); -- raw statement from the 'postgres_test.sql#L9' link above
ERROR: unrecognized parameter "reject_invalid_structures"
-- maybe use parameter mentioned in user manual Oracle?
-- https://lifescience.opensource.epam.com/bingo/user-manual-oracle.html#creating-an-index
CREATE INDEX idx_t_molecule__canonical_smiless_bingo ON t_molecule USING bingo_idx (canonical_smiles bingo.molecule) with (FP_TAU_SIZE='0');
ERROR: unrecognized parameter "fp_tau_size"
-- I know 'REJECT_INVALID_STRUCTURES' and 'FP_TAU_SIZE' can be modified by bingo.bingo_config,
-- but which one control fingerprint type when creating index?
-- It's hard to correspond 'FP_ORD_SIZE', 'FP_ANY_SIZE', 'FP_TAU_SIZE', 'FP_SIM_SIZE' and
-- fingerprint types: 'sim', 'sub', 'sub-res', 'sub-tau', 'full'
select * from bingo.bingo_config;
cname | cvalue
-------------------------------------------+--------
TREAT_X_AS_PSEUDOATOM | 0
IGNORE_CLOSING_BOND_DIRECTION_MISMATCH | 0
IGNORE_CISTRANS_ERRORS | 0
IGNORE_STEREOCENTER_ERRORS | 0
ZERO_UNKNOWN_AROMATIC_HYDROGENS | 0
STEREOCHEMISTRY_BIDIRECTIONAL_MODE | 0
STEREOCHEMISTRY_DETECT_HAWORTH_PROJECTION | 0
REJECT_INVALID_STRUCTURES | 0
IGNORE_BAD_VALENCE | 0
FP_ORD_SIZE | 25
FP_ANY_SIZE | 15
FP_TAU_SIZE | 10
FP_SIM_SIZE | 8
SUB_SCREENING_MAX_BITS | 8
SIM_SCREENING_PASS_MARK | 128
NTHREADS | -1
TIMEOUT | 60000
CT_FORMAT_SAVE_DATE | 1
CT_FORMAT_MODE | AUTO
ALLOW_NON_UNIQUE_DEAROMATIZATION | 1
SIMILARITY_TYPE | sim
(21 rows)
-- my best guess is 'SIMILARITY_TYPE', cuz the default value is 'sim'
UPDATE bingo.bingo_config SET cvalue = 'sub' WHERE cname = 'SIMILARITY_TYPE';
UPDATE 1
CREATE INDEX idx_t_molecule__canonical_smiless_bingo ON t_molecule USING bingo_idx (canonical_smiles bingo.molecule);
ERROR: error: Unknown similarity type 'sub'
UPDATE bingo.bingo_config SET cvalue = 'sub-res' WHERE cname = 'SIMILARITY_TYPE';
UPDATE 1
CREATE INDEX idx_t_molecule__canonical_smiless_bingo ON t_molecule USING bingo_idx (canonical_smiles bingo.molecule);
ERROR: error: Unknown similarity type 'sub-res'
UPDATE bingo.bingo_config SET cvalue = 'sub-tau' WHERE cname = 'SIMILARITY_TYPE';
UPDATE 1
CREATE INDEX idx_t_molecule__canonical_smiless_bingo ON t_molecule USING bingo_idx (canonical_smiles bingo.molecule);
ERROR: error: Unknown similarity type 'sub-tau'
UPDATE bingo.bingo_config SET cvalue = 'full' WHERE cname = 'SIMILARITY_TYPE';
UPDATE 1
CREATE INDEX idx_t_molecule__canonical_smiless_bingo ON t_molecule USING bingo_idx (canonical_smiles bingo.molecule);
ERROR: error: Unknown similarity type 'full'
-- 'SIMILARITY_TYPE' only support "sim", "chem", "ecfp2", "ecfp4", "ecfp6", "ecfp8"
-- https://github.com/epam/Indigo/blob/cc682584c5942742287637817c48eb4669813f8b/api/tests/integration/tests/bingo/similarity_types.py#L45
-- and none of these can create index successfully except "sim".
--: chem
UPDATE bingo.bingo_config SET cvalue = 'chem' WHERE cname = 'SIMILARITY_TYPE';
UPDATE 1
CREATE INDEX idx_t_molecule__canonical_smiless_bingo ON t_molecule USING bingo_idx (canonical_smiles bingo.molecule);
ERROR: error: bingo-postgres: Error while executing build index procedure error: bingo-postgres: molecule build engine: error while processing records: dearomatization: non-unique dearomatization: Dearomatization is not unique. Cannot restore hydrogens. ERROR ON id=245
--: ecfp2
UPDATE bingo.bingo_config SET cvalue = 'ecfp2' WHERE cname = 'SIMILARITY_TYPE';
UPDATE 1
CREATE INDEX idx_t_molecule__canonical_smiless_bingo ON t_molecule USING bingo_idx (canonical_smiles bingo.molecule);
...
NOTICE: bingo.index: 176000 structures processed
ERROR: error: bingo-postgres: Error while executing build index procedure error: bingo-postgres: molecule build engine: error while processing records: Unknown exception
--: ecfp4
UPDATE bingo.bingo_config SET cvalue = 'ecfp4' WHERE cname = 'SIMILARITY_TYPE';
UPDATE 1
CREATE INDEX idx_t_molecule__canonical_smiless_bingo ON t_molecule USING bingo_idx (canonical_smiles bingo.molecule);
...
NOTICE: bingo.index: 70000 structures processed
WARNING: build engine: error while processing record with ctid='(710041,6)'::tid: element: can not calculate implicit hydrogens on aromatic S, charge 0, degree 2, 0 radical electrons
WARNING: build engine: error while processing record with ctid='(710041,9)'::tid: element: can not calculate implicit hydrogens on aromatic S, charge 0, degree 2, 0 radical electrons
WARNING: build engine: error while processing record with ctid='(710023,3)'::tid: element: can not calculate implicit hydrogens on aromatic S, charge 0, degree 2, 0 radical electrons
WARNING: build engine: error while processing record with ctid='(710024,7)'::tid: element: can not calculate implicit hydrogens on aromatic S, charge 0, degree 2, 0 radical electrons
ERROR: error: bingo-postgres: Error while executing build index procedure error: bingo-postgres: molecule build engine: error while processing records: Unknown exception
--: ecfp6
UPDATE bingo.bingo_config SET cvalue = 'ecfp6' WHERE cname = 'SIMILARITY_TYPE';
UPDATE 1
CREATE INDEX idx_t_molecule__canonical_smiless_bingo ON t_molecule USING bingo_idx (canonical_smiles bingo.molecule);
...
NOTICE: bingo.index: 132000 structures processed
ERROR: error: bingo-postgres: Error while executing build index procedure error: bingo-postgres: molecule build engine: error while processing records: Unknown exception
--: ecfp8
UPDATE bingo.bingo_config SET cvalue = 'ecfp8' WHERE cname = 'SIMILARITY_TYPE';
UPDATE 1
CREATE INDEX idx_t_molecule__canonical_smiless_bingo ON t_molecule USING bingo_idx (canonical_smiles bingo.molecule);
...
NOTICE: bingo.index: 154000 structures processed
ERROR: error: bingo-postgres: Error while executing build index procedure error: bingo-postgres: molecule build engine: error while processing records: Unknown exception
```
What is the correct way to change fingerprint type in bingo_idx? Or is it possiable to reset default fingerprint type to 'sub'?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with bingo/postgres/tests/local/postgres_test.sql and the PostgreSQL bingo_idx index path. Compare the documented fingerprint options with the SIMILARITY_TYPE entries in bingo.bingo_config and the observed index-build errors. Done means identifying and documenting the supported way to select the fingerprint type, or confirming that the requested type is unsupported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, postgresql, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100