Pass the YAML configs data to MakeStep._run_step() as they may need it
when running steps.

Signed-off-by: Guillaume Tucker <[email protected]>

Expand the fragment names to their full path in MakeConfig when they
are listed in the YAML configuration.  This allows things like
defconfig+kselftest to be used from a user point of view, as it was
already the case before the refactoring with steps.  Then the expanded
defconfig will have defconfig+kernel/configs/kselftest.config with the
actual path to the fragment file.  Both versions (full and expanded)
are stored in bmeta.json.

Signed-off-by: Guillaume Tucker <[email protected]>

Add the kernel publish_path to the meta-data in MakeConfig.  This will
be needed when pushing build artifacts to the storage server.

Signed-off-by: Guillaume Tucker <[email protected]>

Add kernelci.storage.discover_files() to find files before sending
them to the storage server with upload_files().

Signed-off-by: Guillaume Tucker <[email protected]>

Add kernelci.build.MetaStep as a Step class solely for the purpose of
using the meta-data from the previously saved JSON files.

Signed-off-by: Guillaume Tucker <[email protected]>

Update the "kci_build push_kernel" implementation to use the new
kernelci.build.Step class and associated meta-data.  Also use the code
in kernelci.storage to send the artifacts to drop the dependenty from
kernelci.build.

Signed-off-by: Guillaume Tucker <[email protected]>

Remove the hardcoded '/test' path in the database API URLs as kci_data
is intended to be used to send any kind of data, not just test
results.  Update kci_data and db-configs.yaml accordingly.  Also
update docstrings and copyright notices.

Signed-off-by: Guillaume Tucker <[email protected]>

Rather than providing the path where to send the data, make it part of
the data itself and leave it up to each database implementation to
deal with it accordingly.  Test results now need to wrap all the data
with "test": {}.  Update kernelci_backend accordingly.

Signed-off-by: Guillaume Tucker <[email protected]>

Rather than raw JSON, accept a Python dictionary with the
Database.submit() method.  This drops the dependency on JSON from the
abstract class, so the calling code can construct the data payload
arbitrarily.

Signed-off-by: Guillaume Tucker <[email protected]>

Add a Database.submit_test() method as an alternative entry point to
the generic .submit() method.  Similarly to .submit_build(), the
.submit_test() method is to send one set of test results using the
same schema as previously used with .submit() when it was hard-coded
for test results only.

Signed-off-by: Guillaume Tucker <[email protected]>

To avoid ambiguity, rename the "describe_v" revison field to
"describe_verbose".  This probably should eventually be dropped to
just have "describe", when reviewing the schema.

Signed-off-by: Guillaume Tucker <[email protected]>

Rework how the artifacts.json file is being handled and the schema it
uses so that it can be actually consumed into the database:

* create one entry for each step at the top-level
* add artifacts types: file, directory, tarball
* add a "path" attribute to each entry so they can be found, relative
  to the _install_ directory
* add an optional "key" attribute to be able to find artifacts
* add an optional "contents" attribute as a list of artifacts that can
  be found in directories and tarballs
* update all the parts of the code that add new artifacts accordingly

Signed-off-by: Guillaume Tucker <[email protected]>

Add "steps" and "artifacts" to the bmeta data structure in MetaStep so
it can all be accessed within a single namespace.

Signed-off-by: Guillaume Tucker <[email protected]>

Rather than creating a new JSON data structure for the HTTP POST
request when publishing a build, and then having the backend rely on
the bmeta.json file previously updated to find extra pieces of data,
send the entire meta-data structure.  This depends on the backend code
to be updated accordingly, to validate and parse this data.

Signed-off-by: Guillaume Tucker <[email protected]>

Print any HTTP errors when sending requests to the backend.  This is
particularly useful when the backend is doing the JSON schema
validation and sending errors back to the client.

Signed-off-by: Guillaume Tucker <[email protected]>

Update the job-build.jinja2 Kubernetes template with the new kci_build
commands to build kernels in separate steps.  Generate a local config
file to simplify command line arguments and keep the API token secret
when sending data to the backend.

Signed-off-by: Guillaume Tucker <[email protected]>

When listing all the combinations to build, use the short fragment
name rather than the expanded one.  The expanded one will be
determined later when doing the actual kernel build.

Signed-off-by: Guillaume Tucker <[email protected]>

Fix a mistake in the call to add the build step meta-data which
resulted in the wrong status to be saved.

Fixes: fa346b3 ("kernelci.build: use Step.name in meta-data")
Signed-off-by: Guillaume Tucker <[email protected]>

Add properties and class methods to define the default build output
and install paths.  Use them in the Step() class constructor.  These
can be used by client code to determine where the files will be
created by default.

Signed-off-by: Guillaume Tucker <[email protected]>

Drop the MetaStep class and create the Metadata class instead to hold
all the meta-data implementation independently from the Step
implementation.  Use this class inside Step to handle meta-data.
Client code that needs to read the meta-data can just create a
Metadata class instead of MetaStep which was accidentally also a build
step.

Update all the build steps to use the new Metadata class.  The build
meta-data is now accessible via the 'bmeta' key, similarly to 'steps'
and 'artifact'.  The artifacts are now directly handled exactly like
stored in artifacts.json and an internal representation using a
dictionary is now kept in self._artifacts_map to simplify the logic
when adding artifacts.

Signed-off-by: Guillaume Tucker <[email protected]>

Fix the message about where the build meta-data is being initialised
by using the output path actually defined in the build step rather
than the command line argument which may be None.

Signed-off-by: Guillaume Tucker <[email protected]>

Use the new Step.get_install_path() method to get the kernel build
installation path reliably following the default values.

Signed-off-by: Guillaume Tucker <[email protected]>

Use the new Metadata.get() method rather than get_value() when getting
the whole build meta-data to submit to the database for a kernel
build.

Update the docstrings in kernelci.data.Database accordingly to the new
kernelci.build.Metadata class.

Signed-off-by: Guillaume Tucker <[email protected]>

Add the Args.build_output command line argument to differentiate the
test output directory from the kernel build output one.  This is
primarily useful when reading meta-data from the kernel build output
and writing job definitions in a separate output directory.

Signed-off-by: Guillaume Tucker <[email protected]>

Use the new kenelci.build.Metadata class instead of MetaStep to get
the build meta-data.  Use the new kernelci.build.Step.get_install_path
method to get the kernel installation path using the conventional
default values.

Signed-off-by: Guillaume Tucker <[email protected]>

Align the signatures for the Step.run() method in all the derived
classes using an "opts" argument as a dictionary with arbitrary
options for each step.  This makes the implementation easier to follow
with a cleaner design.

Update kci_build accordingly.

Signed-off-by: Guillaume Tucker <[email protected]>