ScanningWebsiteAvailabilityTool

This process tests the status of internal HREFs given a list of URLs and finds any page errors. Results are summarized in a file that lists all form elements, a site map, and a summary of findings.

Comment 1

S.W.A.T. - Scanning and Website Availability Tool

Business Need

· Automate daily time intensive testing of website page statuses by navigating to webpages via browsers to simulate end-user interaction

· Provide daily confirmation that the site pages are up and running without page errors.

Process

· Reads the URL’s list from URL List file

· Navigates to the URL

· Scrapes the HTML on the page then finds all HREF’s on the page.

· Navigates to all internal HREF’s and repeats scraping process until all internal HREF’s have been scraped.

· Determines whether a page has any server errors and if not then the page is passed.

· An additional scrape is done to list all form input elements on a page if it exists, to determine if all the form elements on the page are displaying correctly.

Output

· Summary of the page status (Number of Pages Passed or Failed, Number of Input Form Elements and Number of Pages found).

· Elements per Page that include the element type and element name.

· Site Map of Webpages.

Future Enhancements

· Determine if a page requires login credentials and completes login process.

· Incorporate multi-tenant processing to improve efficiencies and run times.

· Incorporate more browser types beyond Chrome, including IE, Edge, Safari, and Firefox

· Identify additional website elements including document types (word, pdf, etc.), images, videos.

· Introduce a machine learning component that can perform analysis to determine correct CSS rendering.

Machine Learning Component (Future)

· Currently training a deep learning neural network using an unsupervised learning model to identify what a correctly displayed website looks like.

· Future steps will include deep analysis of the encoded representation learnt by the deep learning model. For instance, we can compress an image of a webpage to a representation understood by the model and use clustering algorithms to make a distinction not only between erroneous and normal webpages, but other features of a website such as text and images.

· This can be integrated into a custom UiPath activity that would query the AI to determine if a webpage rendered correctly as well as what level of certainty.

Built With

Share this project:

Updates