Guys, I found one really weird big in bblfsh. I think it is related to daemon itself but cannot be 100% sure. So do not hesitate to move this issue if I am wrong.

Context

Machine learning team really want to have reproducible results everywhere where it is possible.
So we fix all random seeds to have a deterministic code. Suddenly we find out that two identical runs of style-analyzer quality estimation give different results: src-d/style-analyzer#557. Clues lead to bblfsh and I was able to reproduce the bug without our code. Let me show you.

How to reproduce

Run in bash

docker run -d --rm --name style_analyzer_bblfshd --privileged -p 9432\:9432 \
                bblfsh/bblfshd\:v2.11.0 --log-level DEBUG
docker exec style_analyzer_bblfshd bblfshctl driver install \
                javascript docker://bblfsh/javascript-driver\:v2.6.0
# clone repo with data
git clone https://github.com/laravel/telescope
git checkout 6f0a10ec586cfa1a22218b6778bf9c1572b97912

Also checked for v1.2.0 driver version.

Run in python3

from collections import defaultdict
import pickle
import time
import os
import bblfsh

bc = bblfsh.BblfshClient("localhost:9432")
res = defaultdict(lambda: defaultdict(int))
times = defaultdict(list)
iter_n = 10
for i in range(iter_n):
    for (dirpath, dirnames, filenames) in os.walk("telescope"):
        for filename in filenames:
            try:
                t_start = time.time()
                filepath = os.path.join(dirpath, filename)
                if ".git" in filepath:
                    continue
                resp = bc.parse(filename=filepath)
                if len(res[filepath]) > 0 and resp.status not in res[filepath]:
                    print("iter %.3d New status %d for filepath %s. counter: %s " % (
                        i + 1, resp.status, filepath, dict(res[filepath])))
                res[filepath][resp.status] += 1
                times[filepath].append(time.time() - t_start)
            except Exception as e:
                print(filepath, e)
                res[filepath][repr(e)] += 1

I do not think it is important but bblfsh package version is 2.12.6.

My output head

telescope/resources/.DS_Store Content must be UTF-8, ASCII or Base64 encoded
telescope/resources/.DS_Store Content must be UTF-8, ASCII or Base64 encoded
iter 003 New status 2 for filepath telescope/webpack.mix.js. counter: {0: 2} 
telescope/resources/.DS_Store Content must be UTF-8, ASCII or Base64 encoded
telescope/resources/.DS_Store Content must be UTF-8, ASCII or Base64 encoded
iter 004 New status 2 for filepath telescope/resources/js/routes.js. counter: {0: 3} 
iter 004 New status 2 for filepath telescope/resources/js/base.js. counter: {0: 3} 

How to interpret

Mainly, if you see the message, that some file gets a new status that means that during the identical requests bblfsh return different statuses. For example, for telescope/webpack.mix.js and telescope/resources/js/routes.js it was able to parse file first but not the second time.

Notes

1. If you set iter_n to bigger number you get much more output messages.
2. Sometimes performance for big iter_n become super bad. Can be related to perf: bblfshd 2.9 non responsive after parsing 1000+ files bblfshd#226
3. If you feed only js files it works great and always parse them for iter_n=100.

So it is important for ML team to have reproducible output from bblfsh.
Please tell me if you are able to reproduce the issue or need more info from my side.