Crypto Discords

Much of what I have to share here will mostly relate to crypto discords, where scammers lie in wait for users to ask support questions, and then they pounce to try to steal wallet information. Let’s look at the ways we’ve blocked these folks.

Keeping the bad guys out

We use Sledgehammer to verify new users. Users cannot send a message until they prove theu are human through a CAPTCHA like test:

We also only want accounts that have a verified phone number in their account.

But these two steps alone don’t keep out the scammers.

Role edits

For any role that a user can have, we need to pull permissions: no **private** threads (we had hackers creating threads that the mods could not see to try to scam users). No embedded links. This has to hold for ALL roles that the public can choose. If you miss one, the scammers will find it and exploit it.

(This does mean that some GIF tools do not work – and that sucks, our community has an excellent GIF game.)

We accidentally allowed private threads for one of our language roles – we could not figure out how the scammers were still creating private threads… Until we scouted the setting in the Italian role.

Links

Scammers love to post links to places where they can have conversations outside the eyes of the public (see private threads). Without private threads, they begin posting URLs to scammer discords. So we began blocking URLs (that was a losing battle).

So we started to block keywords “http” and “https” and that blocked *some* of the scammers…. but not all of them.

It turns out Discord “helpfully” clears whitespace from URLs. So this would work as a link.

Yeah – that’s a mess. So we had to build regexes to block every combination of http with various numbers of spaces after every letter. And unicode in the URL? Definitely up to no good. These guys get a 1 week timeout so that we can wake up and ban them.

Aside to any Discord employees who might be reading this – “fixing” URLs to render correctly like this is stupid. You should stop doing that. It will stop MANY MANY scammers in their tracks.

What the scammers do

Anything to get people out of the public eye. If you start chatting in a DM – they have you where they want you.

What do you think? This DM is legit? This guy REALLY wants to help you? Sadly, people CAN and DO fall for these sorts of messages and get scammed. So we created block lists of common terms (and then common misspellings of those terms “sumbit a ticket”, etc. Keeping them from sending these sorts of “hey, Im here to help” type messages.

What are other ways to get into someone’s DMs? Ahh – pretend to be a mod, or leader of the Discord:

Have MOD as their avatar:

(You’d think shithead03244 would be a terrible name for a mod… and you would be correct).

Maybe they’ll attempt to impersonate a mod:

Cousin Dourg:

Does this stop them? Heck no. Once banned – they hang around outside the Discord and have an alt account listening. Then they pounce in with a DM offering help.

So, even with all of this prevention, we constantly have to warn those asking questions that anyone in their DMs is up to no good.

Sometimes, even replying in 3 minutes is not enough. This poor guy got scammed.

If you are running a community Discord, especially a Discord in crypto, I hope some of these tips help you filter out the scammers and the bad actors.

Alice leaves Point A, driving East at 60 kmh...

I know, physics, right? It’s about to get more mathy.

Linear Algebra Vectors

In linear algebra, vectors are one dimensional matrices. If you are a programmer, think array. Each entry has a meaning. We could describe the physics problem above with a 4 entry vector describing Alice’s location (in 2D space) and her speed as:

The first two entries describe her location, and last two describe her motion (speed and direction of travel).

Linear algebra vectors contain data, and can contain a lot of data – there can be a lot of entries in a vector. 

The cool thing about vectors is that when you multiply them together (the dot product), you know the distance between the 2 vectors. By default, the dot product contains the magnitudes of the vectors – and you want the angle between them (cosine similarity), but for this article, dot product will be good enough.

Vectors in Machine Learning: comparing

When a model is trained against millions of objects, vectors are one way that the objects can be described. (Note: It is unclear if ChatGPT works with vectors, but the analogy for their mathematical model is similar.). Because the model can describe any object or sentence as a vector, we can compare the two vectors to get an index

For example, I can ask a LLM model to create a vector of this image:

I can then create text vectors for the terms “dinosaur”, “chicken”, “dog” and “tuna casserole” using the same model.

Each of the vectors created above have 512 entries. This is no longer a dot product for humans to calculate – but easy for a computer. Let’s see what the dot product similarity tells us:

Based on the model we used (I am using Corcel’s APIs, and you can use my Jupyter Notebook to test), the image is most likely a dinosaur, and less likely to be tuna casserole. The model works!

Why good prompts matter

In the above example – when the vectors are compared – the image had more aspects related to the string “dinosaur” than to the string “tuna casserole.”

Now, think about your prompt being turned into a vector to find a match. If you don’t have the context you need – you won’t get the accurate result.

My favorite example around prompt context is the term “football.” In the US, football is a sport with pads, helmets and an oval ball. For the rest of the world, it’s what americans call soccer. (For simplicity, I’ll refer to the two sports as football and soccer for the rest of the post.)

Let’s try image creation prompts:

I love football art. Can you create a comic book style image of football players completing a pass?

This is a mess. This player has soccer shorts, but one arm has football pads. He is catching a soccer ball the way a football player might catch a ball.

Lets refine:

Im from scotland and I love football art. Can you create a comic book style image of football players completing a pass?

Much better. Since in Scotland, football == soccer, we have a drawing of a soccer game.

Let’s try a US location:

Im from texas and I love football art. Can you create a comic book style image of football players completing a pass?

This has strong University of Texas vibes. Another generated image has very strong resemblance to the Dallas Cowboys helmet:

The addition of a location to the image gave a much stronger version of the image – first by removing the football/soccer ambiguity, but it is interesting that it was also able to add a regional context to also adding appropriate football jerseys for the region. (replicated below with a Wisconsin location)

The added context in our prompt is added to the vectors, and it allows the AI to generate responses that are even more appropriate.

Conclusion

Vectors are a common way that LLMs and other machine learning libraries describe items. Vector math can be used to find similarities, and similar contexts for the content being created.

In this post, we’ve covered (at a really high level) how the vectors encode information from the text/images, based on the LLM model. We can the use the vectors with the LLM to create new content, or find similar content.

To conclude, I wanted to share this very funny (to me) image.

Generate an image of a soccer ball next to an american football

I love the stars and stripes on the soccer ball, but it isn’t exactly what I meant

The Chat GPT API – saving you money

To use the GPT-4 model in the ChatGPT app, you need to subscribe at $20/month. And you know, if work is paying for that – this is awesome.

But, I was unemployed for a bit in 2023, and I still wanted that sweet sweet GPT4 content – but that subscription did not make the unemployment cut. The I learned that if you spend $5 in your API account – you get GPT-4 access from the API! So I did it – I spent a week playing and building a few Jupyter Notebooks – and using the API for all my queries, I used up about 15 cents of my API credits. So, if you are using ChatGPT a few times a day, you can save yourself the $240/year, and probably use less than $5 of API calls using GPT4 in an entire year.

Jupyter Notebooks

I have been using Jupyter Notebooks for a few years now, and they have a similarity to the GenAI chat of ChatGPT – so I though it would be an easy way to build my own chat interface. The notebooks are all on Github, so feel free to play around with them.

To use these sample apps, you’ll need Jupyter Labs installed on your computer (install instructions in the Jupyter Notebook link above). The Note book is the Supercharged GPT notebook:

Initializing the chat

The first three cells of the Notebook are just setting the scene:

First, we pull in all the libraries we need, and set our API key (using environmental variables)

#imports all the required libraries, and creates the OpenAI client

from openai import OpenAI
from dotenv import dotenv_values
openaikey = dotenv_values(".env")['openai_apikey']
client = OpenAI(
   api_key=openaikey
)

Next, we initialize all the variables that can be used in the ChatGPT API.

# initial variables that will be used by ChatGPT.  
# Update the startMessages to the context you would like used.  
# This is where you can tell CHatGPT the persona you would like it to take on.

useModel = 'gpt-3.5-turbo'

#use either Temperature or top_p, but not both.  Both default to 1.
useTemp  = 0.2
useTopP = 1
startMessages =[
    {"role": "system", "content": "You are a assistant with a penchant for scientific rigor. Your responses will be verbose but always accurate. If there is uncertainty in your answer, it should be stated explicitly."}
]
useMessages=list(startMessages)
usemaxTokens =None
numberofChoices = 1
useFrequency_penalty = 0
usePresence_penalty = 0

Finally, define two functions: add_chat and reset_chat. after the code, we can talk about what they do.

import json

# Initialize an empty array to store messages
messages = []

#this is the addchat function
def add_chat(role, messageContent):
    # Create a dictionary to represent the chat request
    message = {
        "role": role,
        "content": messageContent
    }
    # Append the JSON object to the 'messages' array
    useMessages.append(message)
    #now make a query to ChatGPT with the new Message array
    response = client.chat.completions.create(
      model=useModel,
      messages=useMessages, 
      temperature = useTemp,
      top_p=  useTopP,
      presence_penalty = usePresence_penalty,
      frequency_penalty=  useFrequency_penalty,
      n=numberofChoices
        
    )
    #when the response comes back, add it t the useMessages array
    newMessage = {"role":response.choices[0].message.role,"content":response.choices[0].message.content}
    useMessages.append(newMessage)
    #print just the chat response
    print(response.choices[0].message.content)
    print(f"This query cost {response.usage.total_tokens} tokens")
    
#reset
def reset_chat():
    global useMessages
    useMessages = []
    useMessages=list(startMessages)
    print("Messages have been reset to start.")
reset_chat()

ChatGPT’s API has 3 types of messages, a system message, a user message and the assistant message. In the 2nd block of code, we use a system message to tell Chat GPT how to reply in all the messages:

startMessages =[
    {"role": "system", "content": "You are a assistant with a penchant for scientific rigor. Your responses will be verbose but always accurate. If there is uncertainty in your answer, it should be stated explicitly."}
]

If we had made our system message, “You are a swashbuckling pirate with a penchant for colorful language” the tone of the replies will change dramatically. But you could

In the add_chat function, we take an input of the message type, and the message. Typically, the message type is user, as we (as the user) are asking GPT a question. We add this message to the existing array of messages, and send the entire conversation to ChatGPT.

When the response comes back, we add it to the message array, but also print it to the user.

In this way, every time we talk to the API – the entire context of the conversation is sent via API, and the response can take that conversation into account.

The reset_chat function does what it says, and clears out the array to begin a new conversation.

And with that, we have a mini chatbot in our Jupyter Notebook!

add_chat("user","what are the max and minimum distances between the earth and the moon? How frequently do they occur?")

Pretty awesome!

Dinkering with parameters

There are a lot of parameters that you can tweak to change the API responses. I think the most common is the temperature – lower values are more consistent, high values produce random results (Accepted values are 0-2).

useFrequency changes how often words can repeat in the response. Allowed values are -2 to 2. Negative values allow for lots of repeating, and some of the results went pretty wacky with weird words showing up over and over.

Its fun to see how modifying the parameters changes the way the responses are created. I remember working on a sound board in high school, and being told not to mess with the knobs above the faders:

The params are like all these buttons and dials – we’re not really sure how they all work, but we can learn how they work through experimentation.

ChatGPT and Hallucinations

Sometimes ChatGPT doesn’t know the answer, and makes mistakes. These are called hallucinations – and I’ve travelled down these rabbit holes in the past. I got some code from ChatGPT – the API endpoint seemed legit, so I worked at my code for a while – only to realize that the endpoint was completely made up!! (argh)

One idea to reduce hallucinations is to run two LLMs side by side. If you ask 2 LLMs the same question, and they both give the same result, it is possible they are both wrong – but it is also likely that the data is accurate. We can add a 2nd LLM to the script, and run them side by side. Then we can feel more assured of the content that comes back.

We’ll use LLama2 as our additional LLM. I installed Llama2 locally on my computer using https://ollama.ai/. Its an easy to use command line interface. Ian Wooten has a great video on how to get started:

In the llama2_chatGPT notebook, I have updated the code from the ChatGPT version, and have inserted the code to make a 2nd call to the Llama 2 instance running locally on my Mac.

Now, when I ask a question, I get two responses!

I feel much more confident in the response when the 2 LLMs agree with one another.

Conclusion

These 2 notebooks were fun ways to learn about the ChatGPT API, some of the parameters we can use to tweak the ChatGOT replies, and better understand how LLMs work. They also had the added bonus of saving some money, while still letting me play with GPTs.

Give them a try, and if they are useful – leave a comment – or drop a note in Twitter (I still go there from time to time). If you find a cool tweak- make a PR to the repository- I’d love to see what you’ve done,a nd share it with the world.

As I apply for a new position here at the end of 2023, I am reading dozens of job descriptions. Each description describes the reporting hierarchy for the developer relations team, and there is a *lot* of variation. Some report into marketing, engineering, customer success, product teams. In smaller organizations, they might report straight right into the CEO. There isn’t any standardization for how developer relations reports inside a company… I don’t necessarily think this is a bad thing.

The reason it isn’t a terrible thing is that *every* devrel position has dotted line reporting into different (every?) parts of the organization.

Dotted Line reporting

Dotted line reporting means that while you directly report into <team name>, you’ll also be receiving tasks, assignments and feedback from *other* teams. Probably every team. This means that, in addition to your goals and expectations from your team, you’ll also be working closely with customer success, product, marketing, engineering, etc., and they will have expectations and goals for developer relations.

Pros of dotted lines

• Learning: By working across silos, I learn more about the issues customers are facing, the engineering pipeline, the marketing plans – you get a good taste of everything that is going on. I can do a better job delivering content to developers!
• Collaboration: Teams that are communicating with one another collaborate better. When everyone is in sync, things can move like a well-oiled machine. When teams are working in a vacuum, work is duplicated, or completely out of sync. Good collaboration keeps everyone in sync.

Cons of dotted lines

• There are a lot of relationships to maintain. It is easy to focus on certain relationships and fall out of sync with others.
• Burnout: If everyone expects work to be completed by the devrel team – the workload can get too high.

While the devrel team may report to one specific member of the team, there is an expectation that there will be many different reporting relationships inside the company – to the point where the DevRel team begins to appear at the center of a web of dotted line reports.

Clearly, some of the dotted lines might be darker and used more often than others, but as a DevRel professional, it is important to keep all lines of communication open and keep the relationships strong.

But wait, There’s more!

In addition to all of the dotted line reporting and collaboration inside your company, the developer relations team is often tasked with building and supporting the developer community.

What does building and supporting a developer community look like? To me, it is building relationships; asking questions; delivering content; creating solutions; providing updates and more. If I were doing these things with an internal team, I’d add a dotted line to my reporting structure.

Are Customers Really a Dotted Line?

What do you think? Leave a comment below!

DevRel acquisition models

In this section, I’ll briefly discuss a couple of popular models that describe the process of making developers aware of your product.

Phil Leggetter introduced the AAARRRP model of the funnel. Here’s his talk at DevRelCon discussing it in detail. I have always visualized this model as follows:

At the top of the funnel, you have those who are “aware” of your product. This is the largest number of users, and we want them to slide down the funnel to become a customer (somewhere in the neck of the funnel is where the customer adds a credit card and becomes a customer).

Another popular model comes from Orbit. The orbit model has a lot of people in your “outer orbit”, and you work to bring them into a closer orbit (and at some point on their journey through your solar system, they become a paying customer).

The ideal model

In the ideal model, the end of the funnel (or the center of our orbit galaxy) would have a black hole:

If this were the case, as soon as a developer becomes aware of our product, they are sucked right through the funnel and become a happy paying customer! Easy peasy, lemon squeezy.

The DevRel Sieve

We all know that the “black hole” model is unrealistic – most users will do a bit more research and ‘kick the tires’ a bit. Every step they take, there is a chance we can lose them. As a result, the funnel is not a funnel, it’s a sieve. Any issue or unclear step and they fall through holes in the funnel, and don’t make it to the end as a paying customer.

What is causing the holes?

Digging through your analytics and usage data can shine some insights into where in the process you are losing developers. Question everything.

• Is the sign-up process too long?
• When a new customer lands in the dashboard, is it empty, or is there a call to action to get started?
• Is there a bad link?
• Is there a poorly worded step in the tutorial?
• Is there a step that can be removed?
• Do your blog posts direct customers to try out the product?
• Do you have a good email follow up process?

If you can find a culprit (or a few culprits), we can then begin proposing fixes to mend the funnel.

Mending the Sieve

As fixes are made to solve the holes in your sieve, you’ll hopefully be able to measure a corresponding increase in customer acquisition.

Lather, Rinse, Repeat.

The process of mending your DevRel funnel is never complete. Some fixes may expose new holes. Iterate over each one, and watch as your customer acquisition numbers improve.

Imaging the process as a developer who isn’t super excited about trying out another tool. See where you get stuck. Is there a confusing step? Now, work to iron it out.

Conclusion

The DevRel team is unlikely to own all of the steps in the developer acquisition funnel. But, they are probably the team with the most holistic view over the entire process. As an advocate for the developer, it is important to make the onboarding process as smooth as possible. And one of the best ways to do that is to plug the holes in your sieve.

Are you looking to improve your Developer Relations funnel? As I write this post (October 2023), I am looking for my next developer relations role. Maybe I can help your team mend their funnel. Reach out, and let’s chat.

Links

Funnel as a sieve isn’t an original idea, but I haven’t seen it applied to DevRel (that I can recall). Here are some posts from a sales point of view:

https://www.sellingpower.com/2010/01/13/3120/are-you-using-a-funnel-or-a-sieve

Your Sales Funnel is a Sieve: Here’s How to Plug the Holes

https://marketinginteractions.typepad.com/marketing_interactions/2014/06/the-b2b-funnel-is-now-a-sieve.html

TL;dr: Stars are a general indicator of Open Source health, but should not be used as a North Star metric.

Github Stars

Every repository on Github can be starred by other users. Some use it as a way to ‘bookmark’ interesting repositories, but it has come to be a metric of how much the repository is used/respected.

Tools like Star-History let you track the star history over time.

Does the number of stars mean something?

Yes. Sort of. If a repository is receiving stars, it is an indication that developers found it useful. If there are stars being given to the repo, that’s great! If the star velocity (the slope of stars/time) is also increasing – that is amazing! It means that the community of developers interested in your project is growing, and that’s a great thing to see.

But, it is just an indicator. Continued growth and accelerated growth are a good indicator that your community is growing, and your project is gaining traction. It is also *very easy* metric to capture – which can become a trap.

Star Games

If stars are a measure of your community, and it is an easy metric to gather, it can quickly take on a life of its own. Rather than focusing on growing the community, the focus becomes on increasing the # of stars

Once the goal is corrupted into “increasing a number on the website” people start playing games.

• Buying Stars: There are websites where you enter your credit card, and within hours/days, your Github repository’s star count increases. (I’m not going to post any links, but if you look, you’ll find them.)
• Shilling for Stars: I love getting swag/t-shirts at conferences. Some companies ask you to fill out a form/ scan a barcode. Some request that you star their Github repo to get the swag.

In both of these cases – your company is spending money (possibly a lot of money) to grow the star metric. In reality, your community hasn’t *really* grown. The ‘stars for swag’ inflate the size of the community, but there is no noticeable velocity change in issues/comments/PRs etc.

A second item to consider – how will you keep your star velocity at the same rate? You have to keep dumping your $$ into the process – for no substantial gain to your community, your codebase, or your product.

What *are* Stars Good for?

When there is a jump in your metrics, you can go back to those dates to discover “what happened.”

I’ve seen large “star bounces” after funding announcements, blog posts, conference talks. In all of these cases, the bounce occurred as a result of the team creating value for developers.

These are the good stars. People who read about your project and go check it out are much more likely to try it out and become a valued member of your community.

Conclusion

Github stars are really easy to measure. Star-history.com builds pretty charts on how the # stars changes over time. But the *right* way to grow your star count involves a team effort and a lot of work!

The goal should be growing your project/growing your community- and the star is an indication that this process is working. If your goal is to increase visibility of your project, the stars will come as developers discover and use your code. But it will take a sustained team effort – building the product/tools and content to get people excited to try out your project.

For me & the CEO? I was able to express these thoughts (perhaps in a much more scattered way), and work to a goal where stars are *important* but not the North Star metric.

If you are a homeowner, you probably have a list of things that need to be done with your house. The house needs a few updates, so one weekend it may be paint or a new dishwasher, new lighting, or replacing planks on the porch. Each of these small tasks makes living in your house better, and improves the overall experience.

Jump in and Try

Developer Relations teams often work in uncharted territory – creating experiments and projects that are new to your company. Can we add X to Y? Could we build a connection with ChatGPT? Building small proof of concept applications is often how neat features get rolled into your product. Bridging your product with other products helps connect communities and raise awareness.

DIY is much the same. The bathroom sink is leaking. So, you head to Home Depot, pick out a new faucet, watch a YouTube video or two, and pull out the wrenches.

In both of these cases, there is a willingness to learn, jump in and try something new. They both will take more time than you expected, troubleshooting (and probably the use of colorful language). But the openness to jump in and try new things leads to a result you can be proud of.

Priorities

Both in DevRel and DIY, there’s a list of things that needs to get done. And the list is is always much longer than the time you have available. So, everything gets prioritized, and you work on the most important stuff first, reshuffling as needed.

But, if an urgent question arises from a customer (or the community), you might put a temporary hold on the “list” and spend a few hours on a document/demo for the customer, and publish the results.

Just like at home… If you have all the walls taped, brushes ready, and the furniture pulled away from the walls to paint the bedroom, but the top step of the porch has come loose – painting will have to wait for a few hours.

Learning and Growing

DevRel teams are tasked with educating the community, but before you can teach others, you have to educate yourself – you have to know what you are talking about in order to be successful. When I started in DevRel, I was technical, but I wasn’t comfortable writing code. But, I jumped in and built sample applications, created demos, all of a sudden I found myself comfortable in that role. Maybe you are super comfortable in JavaScript, but the new demo needs to be in Python or Go. Successful devrel teams jump in and try new things (even if they are uncomfortable at first).

I found the same thing when it came to wring. When it came to electricity – I knew just enough to not zap myself. Then my wife found a really cool light for the wall. I stopped, took some time, watched some videos, and figured out the steps to change out a fixture:

After a few projects working with wiring – like changing out thermostats:

The next thing I knew, I was replacing outlets! (Always be safe, and make sure you shut of power before doing electrical work).

DevRel is technical DIY

If you think about it, we are creating the documentation, code and tutorials to help other developers be successful in their projects. Developers are learning from us in order to build their systems and applications.

Go in as deep as you feel comfortable – if you get stuck – reach out for help. Your dev team probably has encountered similar issues and a quick 10 minute troubleshooting will save you time. Similarly – if you’re not sure about the project, call in for help (Tim has saved us on a number of occasions!)

The things I love about devrel is that every day is different. I find the same enjoyment in working in and around the house.

What else can I do with the same type of process? In this post, we’ll grade the readability of our docs.

Taking the temperature of your Docs

I’d like to use the same idea of iterating through the sitemap of my docs to understand how they are doing. By measuring a few vitals on each page, I can start to piece together the health of the page.

One test I can run on each page to test the page for readability. A readability score uses heuristics to give you an estimate on the grade level required to read the document. Generally they follow:

6-12: High school reading level

12-16/18 College reading level

18-22 graduate school level

22+ Very advanced.

Most importantly, there are Python libraries that will take text and give it a readability grade.

By running a grade level test across all the pages in the docs, we can get a *very* rough idea on how readable your docs are. Reading scores are based on the length of words and sentences. Prose with short sentences and small words is generally considered easier to read. For that reason, sentences like “Your Kubernetes namespace may contain pods with hundreds of directories” will have a higher score than general writing.

When I run the Flesh_kincade grade metrics across the unskript docs and chart across grade level, I see two distinct peaks:

The split is nearly 50:50 at the grade 20 cutoff.

The Runbook also has an interactive table to look at which pages fall into different grade ranges:

Pages with “low” Grade level

When I look at the files <20, they are all documentation pages that have been written by the team that describe issues in prose and with images. These I deem as “generally readable” for a technical audience.

(Of course, we can and will look at the 17-20 grade level docs and see if we can make them easier to read.)

Pages with “high” Grade level

The files that have readability of over 20 are nearly all auto-generated, and fall into 3 categories:

1. Connection instructions: These pages have a screenshot of the connection page, followed by a table that describes each entry field, and the value that should be added. There’s no real advantage to adding prose here – these pages “do their job,” despite scoring poorly on this test.
2. API reference: Have you ever read API docs? Yeah, they are not renowned for readability. These will just score poorly.
3. Action Lists: unSkript has ~500 Drag& Drop actions. To improve visibility of these Actions, we created automated pages that read the name and description of each Action from GitHub. There is a remediation here – we can improve the descriptions in GitHub to improve the readability of these pages.

No articles written by the team appear in this high level list!

Readability as Temperature

When a parent places their hand on their child’s forehead – they are doing a quick check. It doesn’t confirm that a child is sick or not, but it can be an indicator. I look at reading level of a document as an indicator- if the score is high, we should look to see why that is the case, and if we can improve on it.

If you’d like to try this Notebook (or any of the others mentioned in this post, I have published them on Github at https://github.com/dougsillars/devrel-automations. Of course, also read go our docs at https://docs.unskript.com – I’d love your feedback!

So, I spend a lot of my time creating (and writing about) DevOps automations. But I also put on my DevRel hat and do my DevRel-y tasks to help my part of the company grow. And, just like for SRE/DevOps teams, some of these tasks are very repetitive – you might even call them ‘toil’.

Since I am spending a lot of time automating away toil for others, its natural to think about how I might automate away some of the ‘toil’ I have in my own role. I have written about the RunBook I created to collect daily metrics and store them in a central database. But there are several other RunBooks I have created to make my life easier.

404s

So, you’re working in the docs, and moving stuff around, and you accidentally break a link. This happens. We’ve all done it. And a lot of times you catch it. But recently, there was a broken link in our docs that colleague found. We thought – are there tools that can help us find issues like 404s in our docs? I didn’t even bother to do a search – since I figured I could automate this with a Jupyter Notebook pretty quickly.

Build a Jupyter NoteBook

I’m building the automation in a NoteBook using the unSkript framework, so there is an extra cell that initializes unSkript. But the rest of the NoteBook can be run in any Jupyter environment. Here’s the link on GitHub.

The input to this RunBook is a Sitemap.

A sitemap is an XML document that provides information about your site. It is used by search engines to help index your site (and giving the sitemap to Google in the Google Search Console will help your SEO). Every platform I have worked with for docs or websites automatically generates a sitemap for you (and it is generally found at /sitemap.xml.)

Step 1: Read in the Sitemap, and collect a list of the URLs on the page

import requests
import xmltodict
import json

#This Action reads in the Sitemap, converts the XML to a dictionary, 
# and then extracts every URL into a list


response = requests.get(sitemap)
contents = response.text
# Parse the XML data to a dictionary
xml_dict = xmltodict.parse(contents)
#print(xml_dict['urlset'])
urlList = []
for url in xml_dict['urlset']['url']:
    urlList.append(url['loc'])

print("sitemap read in, list of urls created")

The List variable urlList now has every URL extracted from the sitemap.

Step 2:

Let’s loop through each url in urlList (extracted from the sitemap) read in the HTML, and extract every link using the BeautifulSoup library. Then – make a request to each URL from the page and save the HTTP status that is returned, as well as the page where the link was seen. If the response is anything but 200, we know which page has the bad link, and which link is broken!

Note: the docs have a lot of cross-referencing, so if a link has already been checked, we don’t need to check it on a second page. (This does mean that if there is a bad link on multiple pages, it may take a few iterations to find each instance). Also, we exclude all references to localhost and 127.0.0.1, as those urls appear in the docs, but will fail in testing.

import requests
import textstat
from bs4 import BeautifulSoup


urls = urlList
#urls = ["https://unskript.com"]
links = {}

for url in urls:
    # get the text of the file
    response = requests.get(url)
    soup = BeautifulSoup(response.content, "html.parser")
    for link in soup.find_all('a'):
        link_url = link.get('href')
        if (link_url not in links) and (link_url[0:4] == "http") and ("localhost" not in link_url)and ("127.0.0.1" not in link_url) and ("runbooks.sh" not in link_url):
            #print(link_url, url)
            #we want to add it
            link_response = requests.get(link_url)
            link_status = link_response.status_code
            data = {"status": link_status, "first_seen": url}
            links[link_url] = data
print("list completed")

Step 3: List the non-200 responses.

A 200 response means that the link loaded as expected.

If we see anything in the 3xx range, it has moved – and we should update the link to the new page.

A 404 means that the page has not been found, and the link is broken. So we should find the new page, and fix the link. 403 are forbidden – which probably means that the link works, but the page content being blocked by the server from script access (screen scraping and the like will use the same process we are using).

Robot v. Robot

When testing the sitemap from unSkript.com, I received two errors:

https://wellfound.com/company/unskript/jobs 403
https://www.linkedin.com/company/unskript-inc 999

Both of these pages work, but are blocking automations from scraping their pages (one is our Job board, and the other our LinkedIn page. The links are fine. It’s just the robots at these two companies not liking my robot, and blocking it.

Conclusion

Everyone hates broken links. Especially when it is in a site that you are in charge of. Rather than pay for a service to regularly check the links in our docs and on our website, we now have a RunBook that can regularly check for broken links on our site.

Want to try it for yourself? The RunBook is Open Source. It’s currently set to run in the unSkript Automation framework, but if you delete the first cell, it’ll run in any Jupyter environment. And you can be assured that every link on your site is working!

Not so fast

The idea of an open web accessible form anywhere is a great and noble idea. In practice – it doesn’t always pan out. Some websites are blocked from specific locales.- you may have seen a youTube video that says “sorry, this video is not available in your location.” China has “the great firewall” blocking access to many western sites.

That sucks. But it is EVEN worse when it is your bank, or your school, or other site that you need for work that is suddenly blocked. In this post, I’ll show you the workarounds that I used when I was traveling.

VPN

VPNs are pretty widely known and understood. In case you have not heard of a VPN – it makes a virtual connection to another country (of your choice), so that your IP address (a string of digits for your internet connection) appears to be from that country.

If I (currently in the US) VPN into the UK – I appear to be coming from a UK computer, so I can watch the BBC. I can access my UK banks.

At one point during my travels, I was in Serbia, and the website for British Airways didn’t work. I tried everything, a new browser, incognito mode, a cellular hotspot… Finally, I tried a VPN. The issue is that BA.com is blocked in Serbia.

If you’re interested in seeing a VPN in action, you can watch this video:

Sometimes, a VPN does not work – the website blocks VPN traffic too. But you REALLY, REALLY need to access the website. Read on for the Nuclear solution.

The Nuclear Solution: Remote Computer

So you tried a VPN to login to your bank, and you *still* could not get in. Sometimes websites *block* the VPN addresses too. That sort of makes sense: if you are a hacker, and you are blocked, you’ll just VPN to the country and try again.

But, all is not lost. What we can do is create a computer running in your country of choice, login to that computer, and then access the website. Since the traffic is coming from a server inside the country, the traffic is allowed, and you can do what you need to do.

This won’t work for streaming video (Netflix, YouTube) as the refresh rate of the remote desktop is only a few frames per second – and the video will just suck.

Using the Cloud

We are going to set up our server at Amazon Web Services (AWS). AWS hosts some of the biggest websites in the world in datacenters (think a big big building with millions of computers inside). They have a great “free tier” for people to get started. This means we can do this remote login approach for 1 year for free… and then it is ~$12-15 a month afterward… (Or you can just create a new account and start over)

There are a few steps to the process. If you have never use AWS, it can be daunting. there are literally thousands of possible configurations and setups. Rather than enumerate them all here, I have created a video for you to follow along. Here is the rough outline:

1. Create an account at AWS (not shown)
2. Spin up an EC2 server running windows in the country of choice
3. Connect to the computer remotely.
4. Fire up a website and connect from your country.

Honesty time: When I gave my notice to a very large US telecom, they asked that I serve my notice period in the USA. They reminded me that they check all access logs for IP addresses outside the USA.

I was in Romania. I used this trick, and I did not trip any of the warning alerts in the company logins.

I hope these tricks are helpful for anyone traveling abroad for a long period of time and having trouble accessing their websites from home.