Feat: Add CVE support to Snyk datasource by shravankshenoy · Pull Request #1405 · aboutcode-org/vulnerablecode

shravankshenoy · 2024-01-29T13:32:28Z

Fixes #1163

Changes Made

Added function datasource_advisory_from_cve to Snyk data source and other helper functions.

Working

An overview of how the function works is given below

generate_payload_from_cve function returns url for a given cve
Html page from the given cve url is fetched using fetch function
parse_cve_advisory_html function extracts all the vulnerabilities and their snyk_ids and package ids from html page
All the affected versions for the affected package id is obtained using extract_html_json_advisories function
Snyk url and html page are got using generate_advisory_payload and fetch functions
All the data is passed into parse_html_advisory function along with advisory_html page created in step 5, and that will return the final VendorData object with affected and fixed versions

generate_purl is a helper function which generates a purl from package url

Results

Tested the app locally by printing the results yielded for CVE-2024-23641

This matches the result at https://security.snyk.io/vuln?search=CVE-2024-23641

Signed-off-by: Shenoy <[email protected]>

…ed from the cve url Signed-off-by: Shenoy <[email protected]>

Signed-off-by: Shenoy <[email protected]>

shravankshenoy · 2024-01-29T13:39:24Z

Oops looks like one test related to formatting is failing - Black style check failed. Will try to fix that

ziadhany · 2024-01-29T17:09:14Z

Oops looks like one test related to formatting is failing - Black style check failed. Will try to fix that

@shravankshenoy you can just run make valid and check the test again with make test

Signed-off-by: Shenoy <[email protected]>

shravankshenoy · 2024-01-30T11:39:35Z

Thanks @ziadhany . All the tests are passing now.

keshav-space

Thanks, @shravankshenoy. This is a good start.
Please take a look at the suggestions below.

vulntotal/datasources/snyk.py

keshav-space · 2024-02-02T12:48:01Z

vulntotal/datasources/snyk.py

+        pkg_name = package_url_split[1].split(":")[1]
+        namespace = package_url_split[1].split(":")[0]
+
+    elif pkg_type in ("golang", "composer"):


PURL creation for golang needs to be properly handled. Take a look at the example below.

For the snyk url https://security.snyk.io/package/golang/github.com%2Fgoauthentik%2Fauthentik%2Fauthentik%2Fproviders%2Foauth2%2Fviews the corresponding purl would be

PackageURL( type='golang', namespace='github.com/goauthentik/authentik/authentik/providers/oauth2', name='views', )

Thanks for highlighting this. I have modified the code so that namespace for Go packages is captured correctly.

vulntotal/datasources/snyk.py

keshav-space · 2024-02-02T12:59:01Z

vulntotal/datasources/snyk.py

+def generate_purl(package_advisory_url):
+    """
+    Generates purl from Package advisory url.
+
+    Parameters:
+        package_advisory_url: URL of the package on Snyk.
+
+    Returns:
+        A PackageURL instance representing the package
+    """
+    package_advisory_url = unquote_plus(
+        package_advisory_url.replace("https://security.snyk.io/package/", "")
+    )
+
+    package_url_split = package_advisory_url.split("/")
+    pkg_type = package_url_split[0]
+
+    pkg_name = None
+    namespace = None
+    version = None
+    qualifiers = {}
+
+    if pkg_type == "maven":
+        pkg_name = package_url_split[1].split(":")[1]
+        namespace = package_url_split[1].split(":")[0]
+
+    elif pkg_type in ("golang", "composer"):
+        if package_url_split[1] == "github.com":
+            pkg_name = package_url_split[-2]
+            namespace = f"{package_url_split[1]}/{package_url_split[2]}"
+            version = package_url_split[-1]
+        else:
+            pkg_name = package_url_split[-1]
+            namespace = package_url_split[-2]
+
+    elif pkg_type == "linux":
+        pkg_name = package_url_split[-1]
+        qualifiers["distro"] = package_url_split[1]
+
+    else:
+        pkg_name = package_url_split[-1]
+
+    return PackageURL(
+        type=pkg_type, name=pkg_name, namespace=namespace, version=version, qualifiers=qualifiers


We must explicitly handle the PURL creation for each ecosystem from the Snyk URL.
For a better understanding on how to create a PURL for a particular ecosystem, you should check https://github.com/package-url/purl-spec/blob/master/PURL-TYPES.rst

Thanks for sharing this. I have tried my best to make sure the purl-spec is followed properly now, but let me know if I missed something.

vulntotal/tests/test_snyk.py

Signed-off-by: Shenoy <[email protected]>

shravankshenoy · 2024-02-13T14:11:39Z

Thanks a lot for reviewing the code. It not only helped me understand purl-spec much better, but also helped while wrtitng code for some of the other PRs. As a positive side effect of this change, when I ran the code for CVE-2024-23641, it now shows the namespaces, unlike before where namespaces were null.

Let me know if any further changes required, would be glad to the the needful :)

keshav-space

@shravankshenoy see few suggestions below.

keshav-space · 2024-02-15T11:56:24Z

vulntotal/datasources/snyk.py

+    elif pkg_type in ("cocoapods", "hex", "nuget", "pip", "rubygems", "unmanaged"):
+        pkg_name = package_url_split[-1]
+
+    return PackageURL(type=pkg_type, name=pkg_name, namespace=namespace)


Before returning a PURL, make sure that we have the bare minimum type and name available, without these PURL creation will fail. In the event that, for some reason, we are not able to get the type and name, log a warning and skip the advisory for this package.

Done. Added an if condition which logs an error if type or name is missing. Also datasource_advisory_from_cve function has been modified such that if the generate_purl function returns None, the advisory is skipped

keshav-space · 2024-02-15T11:56:35Z

vulntotal/datasources/snyk.py

+    elif pkg_type == "golang":
+        if package_url_split[1] == "github.com":
+            ns_start = package_advisory_url.find("github.com")
+            ns_end = package_advisory_url.rfind("/")
+            namespace = package_advisory_url[ns_start:ns_end]
+            pkg_name = package_url_split[-1]


It is not given that a Go package will always be on GitHub. For instance, Go package https://security.snyk.io/package/golang/blitiri.com.ar%2Fgo%2Fchasquid%2Finternal%2Fsmtp, will not be properly handled by above highlighted code.

Suggested change

elif pkg_type == "golang":

if package_url_split[1] == "github.com":

ns_start = package_advisory_url.find("github.com")

ns_end = package_advisory_url.rfind("/")

namespace = package_advisory_url[ns_start:ns_end]

pkg_name = package_url_split[-1]

elif pkg_type == "golang":

pkg_name = package_url_split[-1]

namespace = "/".join(package_url_split[1:-1])

vulntotal/tests/test_snyk.py

…prove error handling Signed-off-by: Shenoy <[email protected]>

Signed-off-by: Shenoy <[email protected]>

shravankshenoy · 2024-02-20T14:03:04Z

Thanks for reviewing the code. Made all the requested changes.

While working on these changes, I also realized that I had somehow missed on following the purl spec properly. So I have used the dictionary which we get by calling SnykDataSource.supported_ecosystem(), inverted the keys and values, and use the new dictionary to get the purl-spec compliant package type. Hope that works.

keshav-space

Thanks++, LGTM!

* Add function to generate purl from package advisory url Signed-off-by: Shenoy <[email protected]> * Add functions to generate cve url and parse the html advisory generated from the cve url Signed-off-by: Shenoy <[email protected]> * Add function to SnykDataSource class to get advisories from cve Signed-off-by: Shenoy <[email protected]> * Add tests and test data for parse_cve_advisory_html function Signed-off-by: Shenoy <[email protected]> * Add test for generate_purl function Signed-off-by: Shenoy <[email protected]> * Remove to_dict in snyk test file where not required Signed-off-by: Shenoy <[email protected]> * Format code using Black Signed-off-by: Shenoy <[email protected]> * Sort imports correctly using isort Signed-off-by: Shenoy <[email protected]> * Modify generate_purl function in snyk Signed-off-by: Shenoy <[email protected]> * Add additional tests for golang and npm Signed-off-by: Shenoy <[email protected]> * Add non-Github Go packages to test, modify generate_purl function, improve error handling Signed-off-by: Shenoy <[email protected]> * Modify generate_purl to comply with purl spec Signed-off-by: Shenoy <[email protected]> --------- Signed-off-by: Shenoy <[email protected]>

shravankshenoy added 6 commits January 27, 2024 12:59

Add function to generate purl from package advisory url

dd6c5e0

Signed-off-by: Shenoy <[email protected]>

Add functions to generate cve url and parse the html advisory generat…

c85ae9b

…ed from the cve url Signed-off-by: Shenoy <[email protected]>

Add function to SnykDataSource class to get advisories from cve

a4555c8

Signed-off-by: Shenoy <[email protected]>

Add tests and test data for parse_cve_advisory_html function

55ff69a

Signed-off-by: Shenoy <[email protected]>

Add test for generate_purl function

15de639

Signed-off-by: Shenoy <[email protected]>

Remove to_dict in snyk test file where not required

31d09c2

Signed-off-by: Shenoy <[email protected]>

shravankshenoy added 2 commits January 30, 2024 16:54

Format code using Black

db2bb04

Signed-off-by: Shenoy <[email protected]>

Sort imports correctly using isort

c3ba287

Signed-off-by: Shenoy <[email protected]>

shravankshenoy mentioned this pull request Jan 30, 2024

Add CVE support in SnykDataSource #1163

Closed

keshav-space requested changes Feb 2, 2024

View reviewed changes

keshav-space added the VulnTotal Tool for cross-validating vulnerability label Feb 2, 2024

shravankshenoy added 2 commits February 10, 2024 15:25

Modify generate_purl function in snyk

a5ce044

Signed-off-by: Shenoy <[email protected]>

Add additional tests for golang and npm

9d9a1af

Signed-off-by: Shenoy <[email protected]>

keshav-space requested changes Feb 15, 2024

View reviewed changes

shravankshenoy added 2 commits February 18, 2024 11:05

Add non-Github Go packages to test, modify generate_purl function, im…

f23aceb

…prove error handling Signed-off-by: Shenoy <[email protected]>

Modify generate_purl to comply with purl spec

4874cbe

Signed-off-by: Shenoy <[email protected]>

keshav-space approved these changes Feb 22, 2024

View reviewed changes

Merge branch 'main' into feat/vulntotal-snyk-cve-support

484457a

keshav-space merged commit 30d3357 into aboutcode-org:main Feb 23, 2024

pombredanne mentioned this pull request Jul 16, 2024

Import data from OpenSSF Malicious Packages #1412

Closed

Uh oh!

Conversation

shravankshenoy commented Jan 29, 2024

Changes Made

Working

Results

Uh oh!

shravankshenoy commented Jan 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ziadhany commented Jan 29, 2024

Uh oh!

shravankshenoy commented Jan 30, 2024

Uh oh!

keshav-space left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

keshav-space Feb 2, 2024

Choose a reason for hiding this comment

Uh oh!

shravankshenoy Feb 13, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

keshav-space Feb 2, 2024

Choose a reason for hiding this comment

Uh oh!

shravankshenoy Feb 13, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shravankshenoy commented Feb 13, 2024

Uh oh!

keshav-space left a comment

Choose a reason for hiding this comment

Uh oh!

keshav-space Feb 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shravankshenoy Feb 20, 2024

Choose a reason for hiding this comment

Uh oh!

keshav-space Feb 15, 2024

Choose a reason for hiding this comment

Uh oh!

shravankshenoy Feb 20, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shravankshenoy commented Feb 20, 2024

Uh oh!

keshav-space left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shravankshenoy commented Jan 29, 2024 •

edited

Loading

keshav-space Feb 15, 2024 •

edited

Loading