2. UI and YAML Coexistence The biggest blocker is determining how UI-configured schedules and YAML-configured schedules should interact:
YAML in the repo should be the only way to do it. The whole point of this issue/feature request is to persist and version-control the config in Git. UI settings should be disabled immediately, once a YAML config exists in the repo, and then slowly be deprecated. Since the UI consists of filling form fields with crontab syntax, there is not a lot of great UI experience to lose IMHO
1. Configuration Location We don't have consensus on where schedules should be defined:
I would like to see something that has the potential to become compatible with other CI languages like Forgejo Actions (and maybe even Github) in the future.
It's always great to have a bug report contribution be ignored for multiple years, and then see a cleanup-bot asking whether this issue still exists. My volunteered work feels very appreciated. Not.
The ci job fails with the following error (e.g. here):
$ npx playwright install-deps
npm WARN exec The following package was not found and will be installed: playwright
npm WARN EBADENGINE Unsupported engine {
npm WARN EBADENGINE package: '[email protected]',
npm WARN EBADENGINE required: { node: '>=18' },
npm WARN EBADENGINE current: { node: 'v16.4.0', npm: '7.18.1' }
npm WARN EBADENGINE }
npm WARN EBADENGINE Unsupported engine {
npm WARN EBADENGINE package: '[email protected]',
npm WARN EBADENGINE required: { node: '>=18' },
npm WARN EBADENGINE current: { node: 'v16.4.0', npm: '7.18.1' }
npm WARN EBADENGINE }
/root/.npm/_npx/e41f203b7505f1fb/node_modules/playwright-core/lib/server/utils/socksProxy.js:339
static {
^
SyntaxError: Unexpected token '{'
at Object.compileFunction (node:vm:352:18)
at wrapSafe (node:internal/modules/cjs/loader:1025:15)
at Module._compile (node:internal/modules/cjs/loader:1059:27)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1124:10)
at Module.load (node:internal/modules/cjs/loader:975:32)
at Function.Module._load (node:internal/modules/cjs/loader:816:12)
at Module.require (node:internal/modules/cjs/loader:999:19)
at require (node:internal/modules/cjs/helpers:93:18)
at Object.<anonymous> (/root/.npm/_npx/e41f203b7505f1fb/node_modules/playwright-core/lib/remote/playwrightConnection.js:24:25)
at Module._compile (node:internal/modules/cjs/loader:1095:14)
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: exit code 1update playwright image Fixes #5 see e.g. https://gitlab.com/mcnesium/feeds-test/-/jobs/9764492477
mcnesium (7a9a9e31) at 18 Apr 12:26
update playwright image
mcnesium (d2fb0c6c) at 18 Apr 12:15
use a working test example
mcnesium (b7835b5b) at 18 Apr 12:05
update playwright image
Trying to parse Linkedin, it fails at the /authwall page.
Here is the TOML snippet:
[linkedin-linkedin]
title = "Linkedin Test"
url = "https://www.linkedin.com/company/linkedin"
entrySelector = "ul.updates__list li"
titleSelector = "a[data-tracking-control-name='organization_guest_main-feed-card_feed-actor-name']"
linkSelector = "a.main-feed-card__overlay-link"
waitForSelector = "ul.updates__list"and this is what the Gitlab CI output returns, when passing DEBUG=info:
page.waitForSelector: Timeout 30000ms exceeded.
Call log:
- waiting for locator('ul.updates__list') to be visible
- waiting for" https://www.linkedin.com/authwall?trk=bf&trkInfo=AQESgmzRyuBaPQAAAZWUpE8YiQa_HfFLN-uDfAroAVj6SfYqpBqq0QA69rX0WlPmj-rSWFlyd9KL2l2Qz5yPgLtrff0ED4k7mOdmE5Bz4Vu5Uiep3p6Jj8Vd5AMruzAPEwsh6B8=&original_refe…" navigation to finish...
- navigated to "https://www.linkedin.com/authwall?trk=bf&trkInfo=AQESgmzRyuBaPQAAAZWUpE8YiQa_HfFLN-uDfAroAVj6SfYqpBqq0QA69rX0WlPmj-rSWFlyd9KL2l2Qz5yPgLtrff0ED4k7mOdmE5Bz4Vu5Uiep3p6Jj8Vd5AMruzAPEwsh6B8=&original_refe…"
at fetchPageEntries (/root/.npm/_npx/fa12bb8499fb4525/node_modules/feed-me-up-scotty/dist/run.js:128:20)
at async file:///root/.npm/_npx/fa12bb8499fb4525/node_modules/feed-me-up-scotty/dist/run.js:80:33
at async fetchFeedData (/root/.npm/_npx/fa12bb8499fb4525/node_modules/feed-me-up-scotty/dist/run.js:78:25)
at async file:///root/.npm/_npx/fa12bb8499fb4525/node_modules/feed-me-up-scotty/dist/run.js:12:30
at async run (/root/.npm/_npx/fa12bb8499fb4525/node_modules/feed-me-up-scotty/dist/run.js:10:23) {
name: 'TimeoutError'
}I did observe, that when using Httpie in my local terminal to do http https://www.linkedin.com/company/linkedin, it returns the actual requested page HTML.
However, when doing the same with curl https://www.linkedin.com/company/linkedin, it returns a page with some javascript that does window.location.href = "https://" + domain + "/authwall?trk=….
Why is this happening? Seems like that internal Firefox is being redirected like Curl is, while Httpie can somehow sneak around it? Is this something about the HTTP User-Agent? Can it be replaced?
mcnesium (03fcda0d) at 14 Mar 12:31
add linkedin test