Commit 33c1f28
Fix CIS-BP sync: stream ZIP extraction to avoid string length limit
The TF_Information_all_motifs.txt file exceeds Node.js's 512MB string
limit when extracted via adm-zip's getData().toString(). Replace the
in-memory approach with streaming for the web-download path:
- Download ZIPs as fetch streams; convert to Node.js Readable via
Readable.fromWeb() instead of buffering the full arrayBuffer()
- Pipe through unzipper.Parse({ forceStream: true }) to read entries
one at a time without materializing the whole archive
- Parse TF_Information line-by-line with readline so no single large
string is ever allocated
- Buffer individual PWM entries (each ~1 KB) via entry.buffer() —
safe because they are tiny
- Add periodic progress logging every 1,000 stored motifs
- Keep adm-zip path in syncCisbp() for smaller user-uploaded ZIPs
- Install unzipper + @types/unzipper
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>1 parent c549e33 commit 33c1f28
3 files changed
Lines changed: 324 additions & 26 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| 16 | + | |
16 | 17 | | |
17 | 18 | | |
18 | 19 | | |
| |||
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
| 28 | + | |
27 | 29 | | |
28 | 30 | | |
29 | 31 | | |
| |||
0 commit comments