NYCPlanning / NYCPlanning/data-engineering

LION - tweaks to offset point algorithm

Open
#1,758 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

db-cscl
Dominant language
Python
Stars
43
Forks
3
Avg merge
23h 3m
Merged PRs (30d)
44

Description

This doesn't NEED to happen as part of the CSCL project - it sort of explicitly might more be a follow-up as it's really an enhancement. For now, putting it under that umbrella though.

Follow up to #1568, #1642, #1711. Backlog, not in scope of wrapping up POC

In POC, goal was simply recreating production data. However, there are many cases where production does not seem to correctly join the correct atomic polygon due to the offset being too large of a distance.

POC vs Prod

Currently, there is one lot in which the POC differs from prod

Image

The highlighted segment - pictured offset points are from the POC pipeline. There must be some slight difference in implementation of the offset point algorithm. Visually, this could be resolved by decreasing the buffer distance just slightly, but any change effects other segments too

Optimizing/Enhancing

The current algorithm is very performant - while we could go one of the more error-proof methods in https://github.com/NYCPlanning/data-engineering/issues/1568#issuecomment-2790200764, they all share the same failure mode when atomic polygons are very thin and offsets would be on the other side of another segment. And then, they'd also share the same resolution - reduce the offset point to be arbitrarily small.

I ran the pipeline with the current algorithm with an offset of 0.1 feet. There are 124 segments that would change left and/or right atomic polygons

Left

Borough Sequence Number Segment ID dev_left_dynamic_block prod_left_dynamic_block
1 00365 143784 442 428
1 00780 164840 905 901
1 02310 301651 908 101
1 14545 310469 405 401
1 00130 332119 327 179
1 00010 333162 206 201
1 00180 341115 911 916
1 00051 342288 508 222
1 00012 343050 261 267
1 00004 346534 130 148
1 91730 346984 900 914
1 91737 347012 900 916
1 00090 347775 501 103
1 01071 349589 211 203
2 08181 9024017 126 107
2 00020 239913 171 175
2 00027 283673 342 343
2 00023 283756 343 211
2 04035 299195 204 995
2 09522 337133 205 215
2 00066 337307 116 108
2 00062 337312 108 107
2 04768 340963 139 235
2 01640 342420 128 167
2 00010 342525 312 319
2 00010 349820 128 169
2 00040 350171 953 948
2 00020 352373 201 203
2 10120 354356 185 184
3 06150 161303 120 200
3 01011 20067 114 115
3 00072 135585 172 129
3 00035 133954 319 314
3 03984 161224 319 314
3 00135 261803 312 335
3 02333 279232 338 306
3 06645 300199 151 107
3 03005 342723 309 321
3 00028 342760 211 226
3 00410 345651 103 124
4 04860 124912 311 308
4 00750 125943 143 145
4 01850 125596 131 103
4 00394 282033 124 202
4 06010 296445 141 142
4 29047 309532 501 505
4 00113 321501 271 262
4 06044 336269 126 125
4 00240 343171 110 107
4 02151 343777 116 115
4 09990 348137 355 353
4 00090 351267 788 819
5 00020 5820 165 150
5 00012 219876 199 200
5 01065 287855 908 116
5 01050 301804 249 105
5 00040 331805 133 156
5 03546 351347 832 959
5 03543 351365 959 937

right

Borough Sequence Number Segment ID dev_right_dynamic_block prod_right_dynamic_block
1 00082 9004171 321 308
1 01330 239499 204 150
1 01828 243589 431 803
1 01829 243590 431 803
1 01832 243591 431 803
1 00980 261429 930 952
1 00087 302472 321 308
1 10125 343048 261 260
1 00070 345951 112 103
1 01190 346828 201 219
1 02340 346980 900 901
1 02345 347018 900 901
1 01034 349935 924 921
1 01036 349936 924 921
2 03765 139974 108 938
2 01100 9007302 302 304
2 12007 277675 140 139
2 01548 283671 343 342
2 00023 283756 342 166
2 02100 302299 303 135
2 02101 302300 303 103
2 00024 324684 404 403
2 00070 337304 108 116
2 00090 337306 108 117
2 00062 337312 116 115
2 04395 340956 147 236
2 02959 340961 139 154
2 01645 342452 148 132
2 00014 354384 131 196
2 00017 354401 953 973
3 01020 161961 160 181
3 06640 300195 151 103
3 00150 307643 019 032
3 01840 335659 134 104
3 01850 335660 134 104
3 00007 342646 117 108
3 00008 342647 117 108
3 00985 342762 211 244
4 09810 149771 114 110
4 92618 9012520 165 132
4 00002 235793 271 270
4 02105 259698 315 317
4 32090 261177 227 223
4 00400 282024 124 116
4 00061 286689 325 531
4 01050 296441 141 135
4 20721 309052 315 311
4 28040 309500 124 123
4 00237 343169 110 143
4 02160 343775 116 107
4 01446 345276 203 359
4 00005 346728 129 104
4 00122 351265 763 479
4 00090 351267 763 798
4 02155 351269 763 798
4 00046 352007 202 204
4 00047 352198 331 323
5 05330 149196 944 959
5 00095 120587 149 161
5 00010 152692 113 988
5 00042 218476 180 177
5 03190 292569 387 217
5 00010 315761 213 214
5 00010 323681 106 167
5 00380 340575 255 140
5 03549 351356 959 832
5 00130 351541 217 225
5 00030 351542 217 225

And easily copyable segmentids

5820,
20067,
120587,
124912,
125596,
125943,
133954,
135585,
139974,
143784,
149196,
149771,
152692,
161224,
161303,
161961,
164840,
218476,
219876,
235793,
239499,
239913,
243589,
243590,
243591,
259698,
261177,
261429,
261803,
277675,
279232,
282024,
282033,
283671,
283673,
283756,
286689,
287855,
292569,
296441,
296445,
299195,
300195,
300199,
301651,
301804,
302299,
302300,
302472,
307643,
309052,
309500,
309532,
310469,
315761,
321501,
323681,
324684,
331805,
332119,
333162,
335659,
335660,
336269,
337133,
337304,
337306,
337307,
337312,
340575,
340956,
340961,
340963,
341115,
342288,
342420,
342452,
342525,
342646,
342647,
342723,
342760,
342762,
343048,
343050,
343169,
343171,
343775,
343777,
345276,
345651,
345951,
346534,
346728,
346828,
346980,
346984,
347012,
347018,
347775,
348137,
349589,
349820,
349935,
349936,
350171,
351265,
351267,
351269,
351347,
351356,
351365,
351541,
351542,
352007,
352198,
352373,
354356,
354384,
354401,
9004171,
9007302,
9012520,
9024017,

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the current offset point algorithm in the POC pipeline and compare its results with production for the listed segment IDs. The work is done when an agreed algorithm or offset strategy resolves the discrepancies without introducing unacceptable changes to other segments; the issue does not name files or tests to run.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.