Ramblings

More advanced Dataclass sorting

2021-05-14T07:03:00-05:00

More advanced Dataclass sorting

While checking my email this morning I came across another useful Pybites tip.

It was a great little tip, but I wondered, "what if I wanted to sort by bite level?"

Pretty up the output

I know it's just an example code, but I like to make my output a bit more readable. A simple loop of the results is all it takes:

for bite in sorted(bites):
    print(bite)

Output:

Bite(number=11, title='Enrich a class', level=<BiteLevel.ADVANCED: 4>)
Bite(number=21, title='Query a nested DS', level=<BiteLevel.BEGINNER: 2>)
Bite(number=25, title='No promo twice', level=<BiteLevel.INTERMEDIATE: 3>)

Ah, much better!

Using a sort_index

As you can see, the bites are being sorted by their number, but I want to see them sorted by difficulty. The first thing I did was add a sort_index to the dataclass and set it to the property that I wanted to sort by, level:

@dataclass(order=True)
class Bite:
    sort_index: int
    number: int
    title: str
    level: int = BiteLevel.BEGINNER

    def __post_init__(self):
        self.sort_index = self.level

Output:

Traceback (most recent call last):
  File "enum_play.py", line 28, in <module>
    Bite(21, "Query a nested DS"),
TypeError: __init__() missing 1 required positional argument: 'title'

Uh oh! Although it's complaining about a missing title, the problem is actually that it's looking for the sort_index.

Dataclasses field(init=False)

I don't want to initialize that variable, so I make the following change to the Bite class:

from dataclasses import dataclass, field
...

class Bite:
    sort_index: int = field(init=False)
    number: int
    ...

We first import field from dataclasses and then set it to init=False in order to prevent that variable from being initialized.

Bite(sort_index=<BiteLevel.BEGINNER: 2>, number=21, title='Query a nested DS', level=<BiteLevel.BEGINNER: 2>)
Bite(sort_index=<BiteLevel.INTERMEDIATE: 3>, number=25, title='No promo twice', level=<BiteLevel.INTERMEDIATE: 3>)
Bite(sort_index=<BiteLevel.ADVANCED: 4>, number=11, title='Enrich a class', level=<BiteLevel.ADVANCED: 4>)

Um, ok better but I don't want to see sort_index being displayed in the output.

Dataclasses field(repr=False)

In order to prevent sort_index from being displayed, we will have to set the repr to False.

class Bite:
    sort_index: int = field(init=False, repr=False)
    ...

Output:

Bite(number=21, title='Query a nested DS', level=<BiteLevel.BEGINNER: 2>)
Bite(number=25, title='No promo twice', level=<BiteLevel.INTERMEDIATE: 3>)
Bite(number=11, title='Enrich a class', level=<BiteLevel.ADVANCED: 4>)

Much better!

Dataclasses frozen

Looking much better! But what if I wanted to make the dataclass so that it could not be changed in the code? That's where frozen comes in:

@dataclass(order=True, frozen=True)
class Bite:
    sort_index: int = field(init=False, repr=False)
    ...

Output:

Traceback (most recent call last):
  File "enum_play.py", line 25, in <module>
    Bite(11, "Enrich a class", BiteLevel.ADVANCED),
  File "<string>", line 6, in __init__
  File "enum_play.py", line 21, in __post_init__
    self.sort_index = self.level
  File "<string>", line 4, in __setattr__
dataclasses.FrozenInstanceError: cannot assign to field 'sort_index'

Ugh, well at least we know it works!

Setting class variables with setattr

It's complaining because we are setting the index_value in __post_init__. In order to get around it, we'll have to set that value a different way.

...

    def __post_init__(self):
        object.__setattr__(self, "sort_index", self.level)

Output:

Bite(number=21, title='Query a nested DS', level=<BiteLevel.BEGINNER: 2>)
Bite(number=25, title='No promo twice', level=<BiteLevel.INTERMEDIATE: 3>)
Bite(number=11, title='Enrich a class', level=<BiteLevel.ADVANCED: 4>)

There we go!

Conclusion

As you can see, working with dataclasses isn't that hard at all and will save you from having to type too much boilerplate code.

Final code:

from dataclasses import dataclass, field
from enum import IntEnum


class BiteLevel(IntEnum):
    BEGINNER = 2
    INTERMEDIATE = 3
    ADVANCED = 4


@dataclass(order=True, frozen=True)
class Bite:
    sort_index: int = field(init=False, repr=False)
    number: int
    title: str
    level: int = BiteLevel.BEGINNER

    def __post_init__(self):
        object.__setattr__(self, "sort_index", self.level)


bites = [
    Bite(11, "Enrich a class", BiteLevel.ADVANCED),
    Bite(21, "Query a nested DS"),
    Bite(25, "No promo twice", BiteLevel.INTERMEDIATE),
]

for bite in sorted(bites):
    print(bite)

If you would like to learn more about Dataclasses, check out the documentation.

Troubleshooting Connectivity Issues

2020-08-10T11:11:00-05:00

Troubleshooting Connectivity Issues

My son likes to live stream on Youtube and was in the middle of one the other day when his phone ran out of space. He asked me to backup his images and videos for him so that he could make some room.

Some Background

I've been an avid Linux user for ages so I've gotten used to not having to use iTunes. I recently made the move back to Windows 10, but I detest having to use iTunes to do anything on my phone, so I don't even have it installed.

Back when I was on Linux, I had found FTPManager Pro in the Apple app store, which would allow me to start an FTP server so that I could just use Filezilla to upload audio files to my phone as well as allow me to copy my images and videos to my network storage device.

Attacking the Problem

I thought, no problem. I already had Filezilla installed, so I fired it up and connected. Unfortunately, those images and videos did not show up. FTPManager has its own directory that you can access, but Apple apparently won't allow it to access the rest of the filesystem.

My next thought was to just run an FTP server on my computer, and use FTPManager to transfer the files, because from within the app the media files in question are available; just not to its built-in FTP server. So I downloaded Filezilla's FTP server utility. After a quick run through the settings, creating a user and home directory, I fired it up.

Back on the phone, I setup the new server and attempted to connect to it. No-Go. The phone kept reporting that the server was not reachable. I double, tripple-checked all of the settings and everything looked good. I fired up Filezilla on my computer and connected to localhost on port 21 and I was able to get in!

The Problem

Back on the phone, I found a way to diagnose connection issues. Under Network Diagnostics it listed the following entries:

Resolve Host
Ping Host
Connect Port

The Resolve Host was the IP address that I had entered for my machine. I opened up a command prompt on one of my kids computers and attempted to ping my machine. Nothing, 100% packet loss...

So with no connectivity between my phone and my computer and not being able to ping my computer from another computer, it's easy to conclude that the issue is with my computer blocking the traffic.

Firewall: Custom Rule

I remembered that at work, while setting up some Samba mounts on a Windows machine, I had run into similar problems because the server could not ping the machines. I had to go into the Firewall settings and enable ICMP ping on IPv4, we don't use IPv6 so I didn't see a need to enable it on that protocol. I found this really good article on how to do it and was now able to ping my machine in no time.

Firewall: Program Rule

Although FTPManager could now ping my machine, it still could not connect to the FTP server. I went back into the Firewall settings and did the same as above for ICMP, but this time instead of creating a new Custom command, I selected Program.

On the Program screen, I selected This program path and clicked on the Browse:

From within the Open dialog windows, I navigated to where the Filezilla Server was installed and selected it:

From the Action dialog window, I left it at the default Allow the connection:

I left the Profile page as default:

I then gave it a generic name and clicked on Finish:

Here you can see that the new rul is running and is active:

With that final addition, FTPManager was able to connect and I was able to proceeded with the backing up the media files.

Conclusion

Don't give up when you run into problems, there's always a way!

Benchmarking Python Code

2020-05-04T08:56:00-05:00

Need to compare which code is faster, read on to find out how

A friend of mine, Mridu Bhatnagar, has been posting some really cool articles explaining how she's gone about solving some LeetCode challenges. They are all interesting, but due to time constraints, I have only attempted to see if I can do a couple of them.

Move Zeroes

One of the first that I attempted to do was Move Zeroes. I thought that I was being clever, and sure enough, my initial time test seemed to indicate the same. That is, until Suraj Sharma pointed out to me that once the numbers got bigger, that my implementation would be slower.

I decided to see if it was true, and sure enough, once I increased the starting size number, mine was way slower!! At any rate, Suraj really liked how I benchmarked the codes, so I decided to share it here.

I wanted to write a script that would indicate, whose code it was currently running with the results to the right of it. In order to keep it DRY, I decided to use decorators.

Here is what the results look like:

...
Working with 8 zeroes out of 100 numbers
   mridu: 0.0000221729
clamytoe: 0.0000185966
 striker: 0.0000629425
 ...
Working with 10003 zeroes out of 100,000 numbers
   mridu: 8.4343523979
clamytoe: 8.0974295139
 striker: 0.0438303947

Full code can be found in this gist.

Decorator: ID Checker

Part of the requirements for the challenge, was that the same object needed to be modified and not just create a new one. I'm not really sure if this is the way to do it, but I decided that checking to see if the id of the before and after objects were the same was the way to go.

def id_checker(func):
    @wraps(func)
    def wrapper(nums):
        before = id(nums)
        func(nums)
        after = id(nums)

        if DEBUG:
            print(f"{'before':>8}: {before}")
            print(f"{'after':>8}: {after}\n")

        if before != after:
            print("The nums list is not the same!")

    return wrapper

Since our functions all only passed the nums variable, I decided to just hard code it in instead of using *args. In hindsight, I should have used *args. Sometimes while developing code, I like to throw in a DEBUG global variable to turn on/off print statements, so those can be ignored.

Decorator: Timer

I decided that the easiest way to keep track of whose code was being benchmarked, would be to name each of our functions, with our nicknames. Then it would be simple enough to extract the function name from within the code.

To time the functions, I created this timer decorator. It's a pretty straight forward timer. The only unique part is how I extracted the function name.

def timer(func):
    @wraps(func)
    def wrapper(nums):
        start = time()
        func(nums)
        stop = time()
        print(f"{func.__name__:>8}: {stop - start:.10f}")

    return wrapper

If you are not familiar with formatting strings within f-strings, the :>8 simply means to right align it and use 8 spaces. The :.10f just means that it's a float value and that I want it to use 10 digits of accuracy.

Decorating the Functions

Each of the separate functions were decorated in the same manner. To try and keep this discussion short, I'll only illustrate mine.

@id_checker
@timer
def clamytoe(nums: List[int]):
    count = 0
    while True:
        try:
            nums.remove(0)
            count += 1
        except ValueError:
            break

    for _ in range(count):
        nums.insert(len(nums), 0)

The decorators are applied from bottom up. Meaning that the @timer decorator is the first to be ran, followed by the @id_checker. If you are not familiar with decorators, Dan Bader wrote a nice article on them.

Generating the Numbers

Since the size of the number played such a significant part in how the code runs, I decided to create a small little number generator.

def generate_nums(size: int) -> List[int]:
    numbers = [0]
    for _ in range(size - 1):
        numbers.extend(sample(range(10), 1))

    return numbers

I decided to go with list's built in .extend method in order to add the iterable generated by random.sample to my list of numbers.

Test Function

Originally, my intent was to create a "testing"/"benchmarking" function for each piece of code, but ended up being about to run them all from the one. I just forgot to rename it...

def test_one(original):
    nums1 = deepcopy(original)
    nums2 = deepcopy(original)
    nums3 = deepcopy(original)

    mridu(nums1)
    clamytoe(nums2)
    striker(nums3)

    try:
        assert nums1 == nums2 == nums3
    except AssertionError:
        print("Outputs do not match!")
        print(f"original: \n{original}")
        print(f"mridu: \n{nums1}")
        print(f"clamytoe: \n{nums2}")
        print(f"striker: \n{nums3}")

I started off by making copies of the original number and then used the copies to make sure that all three implementations were returning the same numbers.

Benchmarking

Phew, finally getting to kicking it all off! The following bit of code runs the benchmark and generates the report.

if __name__ == "__main__":
    sizes = [10, 100, 1_000, 10_000, 100_000]
    for size in sizes:
        original = generate_nums(size)
        print(f"Working with {original.count(0)} zeroes out of {len(original):,} numbers")
        test_one(original)

I first start with a list of the number lengths that I want to benchmark against. I then iterate over that list and using the generate_nums() function, generate the numbers. This is followed by printing some information of how many zeroes are actually in the number and the length of the number being used. Then finally the actual benchmark is performed.

This is repeated until all of the sizes have been benchmarked.

A Better Way to Benchmark Code

I was working through another one of the challenges that Mridu had completed, Reverse words in a string III. When it came time to compare my implementation with hers, I decided to use the timeit module this time around.

Python comes with the timeit module in the standard library. I don't like that you have to pass the code that you want to benchmark to it as a string, but it works pretty well. It times how long a particular piece of code takes to execute a certain number of times, and returns the fastest time.

Here is the script in it's entirety:

from timeit import timeit

setup_code = """
s = "Let's take LeetCode contest"
"""

mridu = """
def reverseWords(s: str) -> str:
    s = s.split()
    s1 = ''
    for x in range(0, len(s)):
        if x == len(s) - 1:
            s1 += s[x][::-1]
        else:
            s1 += s[x][::-1] + ' '
    return s1


reverseWords(s)
"""

clamytoe = """
def reverse_words(s):
    return " ".join([word[::-1] for word in s.split()])


reverse_words(s)
"""


codes = {
    "clamytoe": clamytoe,
    "mridu": mridu
}

for code in codes:
    print(code, timeit(stmt=codes[code], setup=setup_code))

The results from running it, look like this:

clamytoe 1.9202047239996318
mridu 3.630925637000473

As you can see, in the timeit function, I'm passing stmt and setup variables. The stmt is the actual code to benchmark, while setup is used to setup any code that will be required for it to run properly. The values passed, just need to be strings, so the easiest thing to do is to just enclose your code in triple double quotes and set it to a variable.

You can also pass a number variable that allows you to modify how many times the code is ran, with the default being 1,000,000.

I didn't see a way of using decorators here, so I simply used a dictionary to do the identifying for me.

Summary

I hope that you will now know how to benchmark any of your code. The first method, using time is more suited towards larger code, while timeit is good for shorter snippets of code. The timeit module can do a few more things. I suggest reading up on it if you're curious.

Until next time, keep on coding!

Update

I've always thought that having to enclose your code in triple double quotes was kind of clunky. Well, my buddy Harrison Morgan just demonstrated to me that you don't have to! The same thing can be done with a decorator and a lambda expression!

from functools import wraps
from timeit import timeit


def test_performance(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        result = timeit(lambda: func(*args, **kwargs))
        print(f"{func.__name__:>8}: {result}")

    return wrapper


@test_performance
def mridu(s: str) -> str:
    s = s.split()
    s1 = ''
    for x in range(0, len(s)):
        if x == len(s) - 1:
            s1 += s[x][::-1]
        else:
            s1 += s[x][::-1] + ' '
    return s1


@test_performance
def clamytoe(s):
    return " ".join([word[::-1] for word in s.split()])


if __name__ == "__main__":
    s = "Let's take LeetCode contest"
    clamytoe(s)
    mridu(s)

And the results:

clamytoe: 1.9616992009999876
   mridu: 3.411077960000057

How awesome is that? This is why the Python community is so great! Everyone is willing to freely share in their knowledge and help each other, thanks again Harrison!

Advanced type annotations

2020-04-19T12:20:00-05:00

How to add type annotations to Python dataclasses

During the COVID19 outbreak, a lof of companies are either giving away free books, tutorials, or access their training material. One such company is Pluralsight. For the whole month of April 2020, they have given everyone free access to their complete library! The content is just awesome, some of the best I've seen. I have been working through their Python - Beyond the Basics when I came across their section on Properties and Class Methods.

Sample program

Their sample program was pretty cool:

og-shipping.py

import iso6346


class ShippingContainer:

    HEIGHT_FT = 8.5
    WIDTH_FT = 8.0

    next_serial = 1337

    @staticmethod
    def _make_bic_code(owner_code, serial):
        return iso6346.create(owner_code=owner_code,
                              serial=str(serial).zfill(6))

    @classmethod
    def _get_next_serial(cls):
        result = cls.next_serial
        cls.next_serial += 1
        return result

    @classmethod
    def create_empty(cls, owner_code, length_ft, *args, **kwargs):
        return cls(owner_code, length_ft, contents=None, *args, **kwargs)

    @classmethod
    def create_with_items(cls, owner_code, length_ft, items, *args, **kwargs):
        return cls(owner_code, length_ft, contents=list(items), *args, **kwargs)

    def __init__(self, owner_code, length_ft, contents):
        self.contents = contents
        self.length_ft = length_ft
        self.bic = self._make_bic_code(
            owner_code=owner_code,
            serial=ShippingContainer._get_next_serial())

    @property
    def volume_ft3(self):
        return ShippingContainer.HEIGHT_FT * ShippingContainer.WIDTH_FT * self.length_ft


class RefrigeratedShippingContainer(ShippingContainer):

    MAX_CELSIUS = 4.0

    @staticmethod
    def _make_bic_code(owner_code, serial):
        return iso6346.create(owner_code=owner_code,
                              serial=str(serial).zfill(6),
                              category='R')

    @staticmethod
    def _c_to_f(celsius):
        return celsius * 9/5 + 32

    @staticmethod
    def _f_to_c(fahrenheit):
        return (fahrenheit - 32) * 5/9

    def __init__(self, owner_code, length_ft, contents, celsius):
        super().__init__(owner_code, length_ft, contents)
        self.celsius = celsius

    @property
    def celsius(self):
        return self._celsius

    @celsius.setter
    def celsius(self, value):
        if value > RefrigeratedShippingContainer.MAX_CELSIUS:
            raise ValueError("Temperature too hot!")
        self._celsius = value

    @property
    def fahrenheit(self):
        return RefrigeratedShippingContainer._c_to_f(self.celsius)

    @fahrenheit.setter
    def fahrenheit(self, value):
        self.celsius = RefrigeratedShippingContainer._f_to_c(value)

    @property
    def volume_ft3(self):
        return super().volume_ft3 - RefrigeratedShippingContainer.FRIDGE_VOLUME_FT3


class HeatedRefrigeratedShippingContainer(RefrigeratedShippingContainer):

    MIN_CELSIUS = -20.0

    @RefrigeratedShippingContainer.celsius.setter
    def celsius(self, value):
        if value < HeatedRefrigeratedShippingContainer.MIN_CELSIUS:
            raise ValueError("Temperature too cold!")
        RefrigeratedShippingContainer.celsius.fset(self, value)

As you can see, there is a lot going on in that code.

I don't know how you guys learn, but I just can't sit and watch a video; I have to be actually coding along to make it stick.

That bit of code seemed simple enough, so I decided to kick it up a notch by using dataclasses and type annotations!

Hell week

Hence my journey into hell week commenced!

To make it an even better learning experience, I decided to only read the documentation on type annotations. I lost track of how many hours I spent trying to get it, not only to work, but to get mypy to NOT complain about any part of it.

After a few days of pulling out my hair I relaxed that restriction, but to my surprise, there really wasn't that much documentation on how to do what I wanted to do! Mostly every example out there dealt with type hinting normal classes, not dataclasses!

In the end I discovered that mypy doesn't support type annotations of settable property variables! Oh well, I could not get a clean mypy run in the end, but I definitely learned a lot about type hinting the rest of the code.

What I ended up with

Instead of going step by step over every bit of the code, I'm just going to present it in its entirety and let you soak it in. I'll talk about the parts that gave me the most problems and point out the parts were I learned things.

shipping.py

from dataclasses import InitVar, dataclass, field
from typing import Any, ClassVar, Dict, List, Optional, Tuple, Union

import iso6346


@dataclass
class ShippingContainer:
    owner_code: str
    length_ft: float
    contents: Optional[Union[List[Any], str]]
    HEIGHT_FT: ClassVar[float] = 8.5
    WIDTH_FT: ClassVar[float] = 8.0
    next_serial: ClassVar[int] = 1337
    args: Optional[Tuple[Any]] = field(init=False, repr=False)
    kwargs: Optional[Dict[str, Any]] = field(init=False, repr=False)

    def __post_init__(self):
        self.bic = self._make_bic_code(
            owner_code=self.owner_code, serial=ShippingContainer._get_next_serial(),
        )

    @staticmethod
    def _make_bic_code(owner_code, serial):
        return iso6346.create(owner_code=owner_code, serial=str(serial).zfill(6))

    @classmethod
    def _get_next_serial(cls):
        result = cls.next_serial
        cls.next_serial += 1
        return result

    @classmethod
    def create_empty(cls, owner_code, length_ft, *args, **kwargs):
        return cls(owner_code, length_ft, contents=None, *args, **kwargs)

    @classmethod
    def create_with_items(cls, owner_code, length_ft, items, *args, **kwargs):
        return cls(owner_code, length_ft, contents=list(items), *args, **kwargs)

    def _calc_volume(self):
        return ShippingContainer.HEIGHT_FT * ShippingContainer.WIDTH_FT * self.length_ft

    @property
    def volume_ft3(self):
        return self._calc_volume()


@dataclass
class RefrigeratedShippingContainer(ShippingContainer):
    celsius: float
    _celsius: float = field(init=False, repr=False)
    MAX_CELSIUS: ClassVar[float] = 4.0
    FRIDGE_VOLUME_FT3: ClassVar[float] = 100

    @staticmethod
    def _make_bic_code(owner_code: str, serial: int) -> str:
        return iso6346.create(
            owner_code=owner_code, serial=str(serial).zfill(6), category="R",
        )

    @staticmethod
    def _c_to_f(celsius) -> float:
        return celsius * 9 / 5 + 32

    @staticmethod
    def _f_to_c(fahrenheit) -> float:
        return (fahrenheit - 32) * 5 / 9

    @property
    def celsius(self) -> float:
        return self._celsius

    def _set_celsius(self, value: float) -> None:
        try:
            if value > RefrigeratedShippingContainer.MAX_CELSIUS:
                raise ValueError("Temperature too hot!")
        except TypeError:
            raise TypeError(
                "__init__() missing 1 required positional argumentL 'celsius'"
            )
        self._celsius = float(value)

    @celsius.setter
    def celsius(self, value: float) -> None:
        self._set_celsius(value)

    @property
    def fahrenheit(self) -> float:
        return RefrigeratedShippingContainer._c_to_f(self.celsius)

    @fahrenheit.setter
    def fahrenheit(self, value) -> None:
        self.celsius = RefrigeratedShippingContainer._f_to_c(value)

    def _calc_volume(self):
        return super()._calc_volume() - RefrigeratedShippingContainer.FRIDGE_VOLUME_FT3


@dataclass
class HeatedRefrigeratedShippingContainer(RefrigeratedShippingContainer):
    MIN_CELSIUS: ClassVar[float] = -20.0

    def _set_celsius(self, value: float) -> None:
        try:
            if value < HeatedRefrigeratedShippingContainer.MIN_CELSIUS:
                raise ValueError("Temperature too cold!")
            super()._set_celsius(value)
        except TypeError:
            raise TypeError(
                f"TypeError: __init__() missing 1 required keyword argument: {'celsius'}"
            )

Learning points

Let's start at the top.

The class definitions

shipping.py[Lines 7-17]

@dataclass
class ShippingContainer:
    owner_code: str
    length_ft: float
    contents: Optional[Union[List[Any], str]]
    HEIGHT_FT: ClassVar[float] = 8.5
    WIDTH_FT: ClassVar[float] = 8.0
    next_serial: ClassVar[int] = 1337
    args: Optional[Tuple[Any]] = field(init=False, repr=False)
    kwargs: Optional[Dict[str, Any]] = field(init=False, repr=False)

This class definition is filled with lots of good nuggets.

Class variables must be annotated with ClassVar
contents is not enforced because of the use of the classmethod create_empty, so it was annotated with Optional
args and kwargs were declared because PyCharm was complaining about them not being there, even though the code will run fine without them
args are tuples, not a list!
kwargs are just dicts not a list of them

__post_init__

In this method we generate the custom bic code that's made up from the owner_code, the serial number, and a category code.

shipping.py[Lines 18-22]

    def __post_init__(self):
        self.bic = self._make_bic_code(
            owner_code=self.owner_code, serial=ShippingContainer._get_next_serial(),
        )

Volume property

The @classmethods remained the same, so we're not going to discuss those. Things start to get interesting when it comes to the volume_ft3 property though.

shipping.py[Lines 41-47]

    def _calc_volume(self):
        return ShippingContainer.HEIGHT_FT * ShippingContainer.WIDTH_FT * self.length_ft

    @property
    def volume_ft3(self):
        return self._calc_volume()

As you can see, the logic for the volume calculation was extracted from the property and moved into its own method.

This will come in handy later on when this method gets overwritten in another class that inherits from this one.

Inheriting the main class

Now comes the fun parts! When you normally inherit a class, you usually call super() to initialize some of the variables. Not so with dataclasses! This is one point where I was pulling my hair out to figure out how to declare these and it turned out to be simpler than I thought. Just don't!

shipping.py[Lines 49-55]

@dataclass
class RefrigeratedShippingContainer(ShippingContainer):
    celsius: float
    _celsius: float = field(init=False, repr=False)
    MAX_CELSIUS: ClassVar[float] = 4.0
    FRIDGE_VOLUME_FT3: ClassVar[float] = 100

The part that gave me the biggest headache was declaring the required celcius variable because celsius is also a property value!

We also don't want the user to be able to manually set the celsius value, because it has to be within a set range, so the "private" variable _celsius was created.

shipping.py[Lines 52-52]

    _celsius: float = field(init=False, repr=False)

Pay attention to the usage of field. This tells the class constructor not only to not expect a value to be passed during initialization, but also to not display this variable through repr!

Property init values

Here is how you handle init values that are also properties of the class. A more detailed explanation can be found in this wonderful write-up: Reconciling Dataclasses and Properties in Python

shipping.py[Lines 70-83]

    @property
    def celsius(self) -> float:
        return self._celsius

    def _set_celsius(self, value: float) -> None:
        try:
            if value > RefrigeratedShippingContainer.MAX_CELSIUS:
                raise ValueError("Temperature too hot!")
        except TypeError:
            raise TypeError(
                "__init__() missing 1 required positional argumentL 'celsius'"
            )
        self._celsius = float(value)

As with the property value in the main class, the logic was also extracted from the property and moved into its own method. I also had to add a try/except clause to it because for some reason, if the celcius value was not passed during initialization, it would try and initialize it with a None value but would give some obscure error message...

Overwriting methods

Next we'll skip to overwriting the _calc_volume method.

shipping.py[Lines 96-98]

    def _calc_volume(self):
        return super()._calc_volume() - RefrigeratedShippingContainer.FRIDGE_VOLUME_FT3

Notice how we call the parent class with super() but that we also use the class name while retrieving the class variable FRIDGE_VOLUME_FT3!

Conclusion

There's a lot going on here, and it might take you a while to grok it all, but keep at it and I can assure you that the next time that you need to annotate some dataclasses, this information will come in really handy!

Cleaning up Anaconda environment.yml files

2020-03-24T07:46:00-05:00

Recreating virtual environments with Anaconda

I love working with Anaconda. It makes things so much easier. I enjoy using it so much that I even wrote a guest post article on PyBit.es about my workflow. It goes into much more detail on the subject, but I'll cover a few basics here.

With that being the case, there is one aspect of it that I really hate and that is creating the environment.yml project file for distribution.

What is an environment.yml file

Think of it as the equivalent of python's requirements.txt. With it, anyone can pick up your project and have a working environment up in no time and with all package dependencies resolved!

By convention, just like the requirements.txt file, the file is usually named environment.yml although it can be any name you wish. Services like binder will automatically know that you are using an Anaconda environment when it detects it and automatically generate the environment from it.

How do you use an environment.yml file

Replicating a working virtual environment from the file is relatively simple. If you or the author of the project stuck to conventions and named it environment.yml, then issuing the following command will get you up and running:

conda env create

But what if it's named something else, like for example secret_sauce.yml? in that case you will have to specify the name of the file while using the -f flag, like ah so:

conda env create -f secret_sauce.yml

Anaconda will then create the virtual environment. It will download and install the packages that are listed in the yml file.

How do you create and environment.yml file

So let's say that you want to be able to recreate your current working environment and you want an exact copy, with matching package versions and everything. It's really easy to do, you simply issue the following command:

conda env export > environment.yml

The generated file can be used to recreate your virtual environment exactly if you ever had the need. One draw back is that, in it's current form, it only works for your specific hardware setup!

How to prepare an environment.yml file for mass use

To make it so that others can use it, whether they are on a Windows or a Mac, you will have to prepare the file. In it's current state it looks something like this:

name: dataviz
channels:
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=0_gnu
  - appdirs=1.4.3=py_1
  - attrs=19.3.0=py_0
  - backcall=0.1.0=py_0
  - black=19.10b0=py37_0
  - bleach=3.1.3=pyh8c360ce_0
  - bokeh=2.0.0=py37hc8dfbb8_0
  - bzip2=1.0.8=h516909a_2
  - ca-certificates=2019.11.28=hecc5488_0
  - certifi=2019.11.28=py37hc8dfbb8_1
  - click=7.1.1=pyh8c360ce_0
  ...
  - yaml=0.2.2=h516909a_1
  - zeromq=4.3.2=he1b5a44_2
  - zipp=3.1.0=py_0
  - zlib=1.2.11=h516909a_1006
  - zstd=1.4.4=h3b9ef0a_2
  - pip:
    - geoplotlib==0.3.2
    - pyqt5-sip==4.19.18
    - pyqtwebengine==5.12.1
prefix: /home/mohh/anaconda3/envs/dataviz

NOTE: I've shortened it because it was pretty long...

As I described in my workflow article, you have to remove everything after the second = sign and the prefix: line towards the end.

This particular one had 165 lines!! As you can imagine, that can get pretty tedious. What would any other Python Ninja do in this situation? That's write, hack together a quick little script to do it for you, and that is what I have done.

You can find the script on my gist account here: cleanup-yml.py

cleanup-yml.py:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from pathlib import Path


def clean_yaml(file: Path, skip: int = 5) -> str:
    """Cleans up an Anaconda environment file

    The contents of the file are generated by issuing the following
    command from within an activated Anaconda environment:

        conda env export > environment.yml

    Arguments:
        file {Path} -- Path file object that contains the exported env
        skip {int} -- Number of lines to skip/leave alone at the top
                      of the file

    Returns:
        str -- cleaned up data
    """
    cleaned = ""
    with file.open() as f:
        for n, line in enumerate(f.readlines()):
            if n < skip or "==" in line or "pip:" in line:
                cleaned += line
            elif line.startswith("prefix:"):
                cleaned += "\n"
            elif "=" not in line:
                cleaned += line.rstrip()
            else:
                keep, _ = line.rsplit("=", 1)
                cleaned += f"{keep}\n"
    return cleaned


def save(file: Path, data: str):
    """Saves the data to the file Path object

    Arguments:
        file {Path} -- Path file object to save the data to
        data {str} -- The data to save to the file object
    """
    file.write_text(data)


if __name__ == "__main__":
    file = Path("environment.yml")
    parsed = clean_yaml(file)
    save(file, parsed)

How to use the cleanup-yml.py script

Using the script is simple. I like to put my scripts into the ~/bin/ directory that I have in my home folder. Since that directory will automatically be in my path, I will be able to run the script from anywhere on my system. Make sure to make the script executable: chmod +x ~/bin/cleanup-yml.py

To use it, you must run it from your project's home directory so that it can find the environment.yml file. There is no output; I guess I should have added a message stating that the modification were completed... A typical workflow would be as follows once you've setup your virtual environment by hand:

conda env export > environment.yml
cleanup-yml.py
pip freeze > requirements.txt

How to create an environment.yml without version lock in

If you don't really care about having an exact replica of your environment. Meaning that you don't care about the package versions, and you just want the environment with the latest version of the packages installed.

This command is what you want:

conda env export --from-history > environment.yml

This will generate an a yml file with just the packages that you specified and non of their dependencies.

Here's an example one:

name: dataviz
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.7
  - pandas
  - seaborn
  - jupyterlab
  - squarify
  - bokeh
  - matplotlib
  - pyglet
  - numpy
  - flake8
  - isort
  - black
  - pytest
  - pytest-cov
  - mypy
prefix: /home/mohh/anaconda3/envs/dataviz

For that one, you would just need to remove the prefix line at the end before distributing.

Conclusion

Writing that cleanup script is really going to improve my workflow, and I hope that it will do the same for yours!

In normal Python, using the requirements.txt file, you would first create your virtual environment, activate it, and then install the packages with the requirements.txt file. With Anaconda, it's all done in one step. You also have the option of choosing different versions of Python, plus you get the guarantee that there will be no package conflicts.

On a side note, it feels great to write a script, with type hinting to boot, and not have black, flake8, isort, or mypy complain about any of it! Sure helps to shake that importer syndrome a bit...

Bash script into a Makefile

2020-03-17T05:08:00-05:00

Using make can literally make your life easier

So the other day on the Pybites slack channel, Erik O'Shaughnessy and I were chatting about something and I happened to mention that I had written a bash script to generate documents from Asciidoc files but wanted to create a Makefile to do the same instead. Without hesitation, he asked for my bash script and got to work! I don't think that I could have found a better tutorial on the subject if I had looked. I've looked things up before and nothing is as succinct as what he wrote up for me.

This is all his work and even though I made some changes, the credit for it all goes to him. I thought it was so great, that I had to share it with everyone because I believe that it can help others get up to speed with creating Makefiles.

What is a make and what is a Makefile

The GNU Make project's page describes it as such:

GNU Make is a tool which controls the generation of executables and other non-source files of a program from the program's source files.

It basically means that you can automate some common tasks with it. To illustrate what it can do, I am going to show you how Erik converted my bash script.

The bash script

Here is the script in all its glory... It's not much. It was just thrown together pretty quick and it was getting the job done.

gen_book.sh

#!/bin/bash
DOC='sop'
DATE=`date +%Y-%m-%d`
REV=`grep revnumber sop.asc | cut -d ' ' -f 2`
PARAMS="--attribute revnumber=$REV --attribute revdate=$DATE"

echo "Converting to HTML..."
bundle exec asciidoctor $PARAMS $DOC.asc
echo " -- HTML outpt at $DOC.html"

echo "Converting to ePub..."
bundle exec asciidoctor-epub3 $PARAMS $DOC.asc
echo " -- ePub output at $DOC.epub"

echo "Converting to Mobi (kf8)..."
bundle exec asciidoctor-epub3 $PARAMS -a ebook-format=kf8 $DOC.asc
echo " -- Mobi output at $DOC.mobi"

echo "Converting to PDF ... (This one takes a while)"
bundle exec asciidoctor-pdf $PARAMS $DOC.asc
echo " -- PDF output at $DOC.pdf"

Creating the Makefile

Variables

This is how you assign a string to a variable name in Make. The variable name doesn't have to be all caps and the equal doesn't have to be snugged up to the variable name; it's just how I write Makefiles

FILENAME= sop

Referencing variables

$(IDENTIFIER) is how you reference a variable in Make, you can use ${} too, but I prefer $(). If you forget the parentheses or the curly braces, eg $INDENTIFER, Make will interpret that as $I with NDENTFIER appended to it. Probably not what you are expecting.

SOURCE= $(FILENAME).asc

HTML= $(FILENAME).html
PDF= $(FILENAME).pdf
EPUB= $(FILENAME).epub
MOBI= $(FILENAME).mobi

Shell commands

$(shell shell command ) is how you invoke a command and save the results to a variable. It gets kinda tricky since every time you reference $(DATE) it will execute the command. The weird assignment operator := means just assign it one time.

DATE := $(shell date +%Y-%m-%d)

Again, calling shell. This time using AWK to pull out the revision number.

The $ needs to be doubled in the command string to keep Make from trying to expand $2 into something we didn't intend.

I know there was a grep|cut in the bash version, I prefer to use awk for these kinds of snip operations since it's a single process invocation. Those are easier to deal with in this context since you don't have to worry about any pipe weirdness imposed by Make.

REVISION := $(shell awk '/revnumber/ {print $$2}' $(SOURCE))

Functions / macros

Ok this is a "function" definition that we use to build the various ASCIIDOCTOR invocations. We could have just written the format specific definitions:

ADOC_HTML= bundle exec asciidoctor
ADOC_PDF= bundle exec asiidoctor-pdf
...

The advantage of this technique is you only have to change the BUNDLE_EXEC part if the way you invoke asciidoctor changes (I don't know why it would change, but the idea is to isolate stuff that's repeated so you don't have to freaking change it everywhere).

Macro or function definition

BUNDLE_EXEC= bundle exec $(1)

Using the macro

ASCIIDOCTOR_HTML= $(call BUNDLE_EXEC,asciidoctor)
ASCIIDOCTOR_PDF=  $(call BUNDLE_EXEC,asciidoctor-pdf)
ASCIIDOCTOR_EPUB= $(call BUNDLE_EXEC,asciidoctor-epub3)
ASCIIDOCTOR_MOBI= $(call BUNDLE_EXEC,asciidoctor-epub3)

Shared flags

Here we build the shared flags used by asciidoctor by all invocations. I use the += assignment to show how you can add to a variable after it's initial assignment.

ADOC_FLAGS= --attribute revnumber=$(REVISION)
ADOC_FLAGS+= --attribute revdate=$(DATE)

Phony rule

This next bit is some make magic. The .PHONY rule is how we tell make that some of our rules are not associated directly with a file.

The all rule below is by default a dependency of .PHONY

.PHONY: html pdf epub mobi

Default rule

The default rule that Make looks for when invoked as make is all. To build the all target, make builds, replace all with "a rule" name.

rulename: dep1 dep2 dep3 ... depN
command_0
@command_1
...
comand_n

dep1: subdep1 ...

Here, the dependencies for all are $(HTML), $(PDF), $(EPUB) and $(MOBI) which expand in to the names of the files that asciidoctor will create. So make all will run the HTML, PDF, EPUB and MOBI rules in that order.

all: $(HTML) $(PDF) $(EPUB) $(MOBI)

Rules

The $(HTML) rule depends on $(SOURCE), and only executes if the source file has changed or the destination file does not exist. $@ is an alias for the name of the rule to be used in the body of the recipe.

By default, make will print the command that is being executed to stdout followed by it's output. To suppress printing the command, preface the command with an @.

Lastly, the indention is a TAB and not 8 spaces. Make is an old-school tool and will complain if it doesn't get it's tabs.

$(HTML): $(SOURCE)
    @echo Converting $(SOURCE) to $@
    @$(ASCIIDOCTOR_HTML) $(ADOC_FLAGS) $(SOURCE)

$(PDF): $(SOURCE)
    @echo Converting $(SOURCE) to $@
    @$(ASCIIDOCTOR_PDF) $(ADOC_FLAGS) $(SOURCE)

$(EPUB): $(SOURCE)
    @echo Converting $(SOURCE) to $@
    @$(ASCIIDOCTOR_EPUB) $(ADOC_FLAGS) $(SOURCE)

$(MOBI): ADOC_FLAGS += -a ebook-format=kf8
$(MOBI): $(SOURCE)
    @echo Converting $(SOURCE) to $@
    @$(ASCIIDOCTOR_MOBI) $(ADOC_FLAGS) $(SOURCE)

NOTE: The MOBI asciidoctor command requires additional flags to generate the file. This is how you would make the changes to it.

At this point, you can add each of the PHONY rules. You might want to group these together along with the regular rules for keep related blocks of logic together, but I'll add them here for now.

html: $(HTML)
pdf: $(PDF)
epub: $(EPUB)
mobi: $(MOBI)

The debug rule

The debug rule is how I checked to make sure all of the variables I constructed contained the things I thought they should.

debug:
    @echo Rule -> $@
    @echo '          SOURCE: $(SOURCE)'
    @echo '        REVISION: $(REVISION)'
    @echo '            HTML: $(HTML)'
    @echo '             PDF: $(PDF)'
    @echo '            EPUB: $(EPUB)'
    @echo '            MOBI: $(MOBI)'
    @echo '       DOC_FLAGS: $(ADOC_FLAGS)'
    @echo 'ASCIIDOCTOR HTML: $(ASCIIDOCTOR_HTML)'
    @echo ' ASCIIDOCTOR PDF: $(ASCIIDOCTOR_PDF)'
    @echo 'ASCIIDOCTOR EPUB: $(ASCIIDOCTOR_EPUB)'
    @echo 'ASCIIDOCTOR MOBI: $(ASCIIDOCTOR_MOBI)'

The clean rule

Often times we want to restart from a known good "clean" state. A clean rule is a good place to remove transient files so you ensure that all your dependencies in your project are rebuilt. In this case we just remove the translated files. We could use wildcards in this rule like rm *.html but this can have unintended consequences if we have other files in HTML format that we didn't want to smoke.

Always be as explicit as possible in clean rules.

clean:
    @/bin/rm -f $(HTML) $(PDF) $(EPUB) $(MOBI)

The help rule

I've found it to be very useful for letting users know what commands are available and what they do. Mostly it's for me, I forget everything...

help:
    @echo 'Makefile for generating documents from Asciidoc source files              '
    @echo '                                                                          '
    @echo 'Usage:                                                                    '
    @echo '   make                                runs rules specified under all     '
    @echo '   make all                            generates all of the file formats  '
    @echo '   make clean                          remove the generated files         '
    @echo '   make debug                          prints all of the variables used   '
    @echo '   make epub                           (re)generates an epub file         '
    @echo '   make help                           prints this message                '
    @echo '   make html                           (re)generates an html file         '
    @echo '   make mobi                           (re)generates a mobi file          '
    @echo '   make pdf                            (re)generates a pdf file           '
    @echo '   make -n [epub, html, mobi, pdf]     prints out the commands it would   '
    @echo '                                       run without executing them         '
    @echo '                                                                          '

Conclusion

Make is a very versatile program and can be used for many more things. For instance Erik uses it to launch his kids Minecraft server! Now that I know how a Makefile is constructed, I will be able to automate other parts of my daily tasks!

NOTE: The MOBI format kept failing on me. It would leave behind a file with the extension -kf8.epub and never generate the .mobi one. My guess is that I need to install some kind of Amazon Kindle app program or something, so I removed it from the all rule to prevent it from generating by default.

This is the final file:

# Makefile

# Main project file name
FILENAME= pnc-sop

# Variables used
SOURCE= $(FILENAME).asc

HTML= $(FILENAME).html
PDF= $(FILENAME).pdf
EPUB= $(FILENAME).epub
MOBI= $(FILENAME).mobi

# Store the current date
DATE := $(shell date +%Y-%m-%d)

# Grab revision number from the source document
REVISION := $(shell awk '/revnumber/ {print $$2}' $(SOURCE))

# Macro or 'function' definition
BUNDLE_EXEC= bundle exec $(1)

# Assigning the macro
ASCIIDOCTOR_HTML= $(call BUNDLE_EXEC,asciidoctor)
ASCIIDOCTOR_PDF=  $(call BUNDLE_EXEC,asciidoctor-pdf)
ASCIIDOCTOR_EPUB= $(call BUNDLE_EXEC,asciidoctor-epub3)
ASCIIDOCTOR_MOBI= $(call BUNDLE_EXEC,asciidoctor-epub3)

# Shared flags
ADOC_FLAGS= --attribute revnumber=$(REVISION)
ADOC_FLAGS+= --attribute revdate=$(DATE)

# Enable phony rules
.PHONY: html pdf epub mobi

# Define the make commands
all: $(HTML) $(PDF) $(EPUB)

# Define each of the commands and specifying their outputs
$(HTML): $(SOURCE)
    @echo Converting $(SOURCE) to $@
    @$(ASCIIDOCTOR_HTML) $(ADOC_FLAGS) $(SOURCE)

html: $(HTML)

$(PDF): $(SOURCE)
    @echo Converting $(SOURCE) to $@
    @$(ASCIIDOCTOR_PDF) $(ADOC_FLAGS) $(SOURCE)

pdf: $(PDF)

$(EPUB): $(SOURCE)
    @echo Converting $(SOURCE) to $@
    @$(ASCIIDOCTOR_EPUB) $(ADOC_FLAGS) $(SOURCE)

epub: $(EPUB)

$(MOBI): ADOC_FLAGS += -a ebook-format=kf8
$(MOBI): $(SOURCE)
    @echo Converting $(SOURCE) to $@
    @$(ASCIIDOCTOR_MOBI) $(ADOC_FLAGS) $(SOURCE)

mobi: $(MOBI)

# Use debug rule to check that all of the variables were
# constructed properly.
debug:
    @echo Rule -> $@
    @echo '          SOURCE: $(SOURCE)'
    @echo '        REVISION: $(REVISION)'
    @echo '            HTML: $(HTML)'
    @echo '             PDF: $(PDF)'
    @echo '            EPUB: $(EPUB)'
    @echo '            MOBI: $(MOBI)'
    @echo '       DOC_FLAGS: $(ADOC_FLAGS)'
    @echo 'ASCIIDOCTOR HTML: $(ASCIIDOCTOR_HTML)'
    @echo ' ASCIIDOCTOR PDF: $(ASCIIDOCTOR_PDF)'
    @echo 'ASCIIDOCTOR MOBI: $(ASCIIDOCTOR_MOBI)'
    @echo 'ASCIIDOCTOR EPUB: $(ASCIIDOCTOR_EPUB)'

# Simple help menu showing what commands are available
# and what they do.
help:
    @echo 'Makefile for generating documents from Asciidoc source files              '
    @echo '                                                                          '
    @echo 'Usage:                                                                    '
    @echo '   make                                runs rules specified under all     '
    @echo '   make all                            generates all of the file formats  '
    @echo '   make clean                          remove the generated files         '
    @echo '   make debug                          prints all of the variables used   '
    @echo '   make epub                           (re)generates an epub file         '
    @echo '   make help                           prints this message                '
    @echo '   make html                           (re)generates an html file         '
    @echo '   make mobi                           (re)generates a mobi file          '
    @echo '   make pdf                            (re)generates a pdf file           '
    @echo '   make -n [epub, html, mobi, pdf]     prints out the commands it would   '
    @echo '                                       run without executing them         '
    @echo '                                                                          '

# Specify clean-up rules.
clean:
    @/bin/rm -f $(HTML) $(PDF) $(EPUB) $(MOBI) $(FILENAME)-kf8.epub

Thanks again Erik! Check out his commented Makefile for further insights!

Testing ABC's with abstract methods with Pytest

2020-03-12T10:31:00-05:00

So you want to test ABC's with pytest

I was working on writing a new code challenge for Codechalleng.es the other day. It's based on inheritance and composition and uses Abstract Base Classes to define the interfaces that should be implemented in classes derived from it.

I was just finishing up with the tests and everything was passing. Then when I checked the code coverage on it, I saw that it was complaining about my Site class, which is the one derived from ABC has several methods decorated as abstractmethods. Coverage wanted me to test the pass statements on them...

Initial searches didn't turn up much, so I posed the question in PyBites slack channel. The best suggestion was to use the dis module and test it that way, but since it's not really running the code, coverage still counts it as untested.

How to test abstract methods

Finally I came across this post on reddit that had the solution. Here is what the code looks like, with doctrings removed:

class Site(ABC):
    web: Web

    def find_table(self, loc: int = 0) -> str:
        return self.web.soup.find_all("table")[loc]

    @abstractmethod
    def parse_rows(self, table: Soup) -> List[Any]:
        pass

    @abstractmethod
    def polls(self, table: int = 0) -> List[Any]:
        pass

    @abstractmethod
    def stats(self, loc: int = 0):
        pass

Here is how you go about testing them:

def test_site(test_file):
    Site.__abstractmethods__ = set()

    @dataclass
    class Dummy(Site):
        web: Web

    url = "https://projects.fivethirtyeight.com/polls/"
    test_web = Web(url, test_file)
    d = Dummy(test_web)
    table = d.find_table()
    rows = d.parse_rows(table)
    polls = d.polls()
    stats = d.stats()
    assert d.web.file.name == "test.html"
    assert isinstance(Site, ABCMeta)
    assert rows is None
    assert polls is None
    assert stats is None

As you can see, the way to accomplish this is to override __abstractmethods__. Basically tricking it into thinking that it doesn't have any. The last three asserts then verify that None is being returned.

============================= test session starts ==============================
platform linux -- Python 3.7.3, pytest-4.4.0, py-1.8.0, pluggy-0.9.0 -- /home/mohh/anaconda3/envs/pybites/bin/python
cachedir: .pytest_cache
rootdir: /home/mohh/Projects/team-charlie/020
plugins: cov-2.8.1, asyncio-0.10.0
collected 8 items                                                              

test_composition.py::test_file_class PASSED                              [ 12%]
test_composition.py::test_web PASSED                                     [ 25%]
test_composition.py::test_web_bad_url PASSED                             [ 37%]
test_composition.py::test_poll PASSED                                    [ 50%]
test_composition.py::test_leaderboard PASSED                             [ 62%]
test_composition.py::test_rcp_stats PASSED                               [ 75%]
test_composition.py::test_nyt PASSED                                     [ 87%]
test_composition.py::test_site PASSED                                    [100%]

----------- coverage: platform linux, python 3.7.3-final-0 -----------
Name             Stmts   Miss  Cover   Missing
----------------------------------------------
composition.py     107      0   100%


=========================== 8 passed in 7.22 seconds ===========================

Success!

Conclusion

If you're ever faced with having to write test for abstract methods, simply override the __abstractmethods__ of the ABC class! Definitely something to keep in your toolbox.

Until next time, happy coding!

Playing around with Jupyter notebooks

2020-03-09T19:43:00-05:00

Seeing how notebooks compare to just plain Markdown¶

What is a Jupyter Notebook¶

Well it's a document that lets you render markdown code, display code blocks, and also allows you to execute those code blocks and display their results interactively! Needless to say, they are definitely game changers. It's like the Python interpreter on steroids!

They are perfect for academia, data scientist, and anyone else that wants to explore data and code and be able to reproduce and/or show how they did it all in the same document. There is a large movement in the scientific community to include the code that was used on research papers, and Notebooks are filling that niche perfectly!

Update to my newpost script¶

To show a little bit about what Notebooks can do, I'll demonstrate by talking about my newpost.py script. I had orginally written it to create the basic structure of new blog posts, but in Markdown. Since I'm going to start using Jupyter Notebooks as well, I decided to spruce it up a bit and add support for them. I want to keep my notebooks, vanilla. Meaning that I don't want to shove the metadata required by Pelican into them. Since I didn't know how much code went into a blank notebook, I decided to just dump the metadata into an .nbdata file. All that's left for me to do is to fire up JupyterLab, create a new book, and name it exactly like the .nbdata file. I didn't want to bother with typing it all out, so I used Pyperclip to copy the slug name into my clipboard. Now a simple paste gets the job done!

Imports and constants¶

The first thing any Python scripts starts with is importing all of the libraries that are going to be used at the beginning of the file, followed by any global variables.

In [14]:

from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from typing import List

import pyperclip

AUTHOR = "Martin Uribe"
ARTICLE_LOCATION = Path("content", "articles")

As you can see, that looks like any other code block that you normally see on blog posts, but this one is actually "active". Let me demonstrate. The following will be a simple Python print() command, and the output that follows will be generated and not typed in by me: [on/off]

In [18]:

print(f"{AUTHOR.split()[0]} is storing his blog articles in {ARTICLE_LOCATION.absolute()}")

Martin is storing his blog articles in /home/mohh/Projects/ghpages/content/articles/content/articles

Pretty nice, right?

Article class¶

I think you're starting to see how powerful working in notebooks really is. Let's continue with the rest of the script. Here is the Article class that I created for it: [on/off]

In [19]:

@dataclass
class Article:
    title: str
    category: str
    tags: List[str]
    summary: str
    markdown: bool

    def __post_init__(self):
        self.category = self.category.title()
        self.date = datetime.today().strftime("%Y-%m-%d %H:%M")
        self.slug = self.title.lower().replace(" ", "-")
        self.author = AUTHOR
        if self.markdown:
            self.location = ARTICLE_LOCATION.joinpath(f"{self.slug}.md")
        else:
            self.location = ARTICLE_LOCATION.joinpath(f"{self.slug}.nbdata")

    def create_file(self):
        header = f"title: {self.title}\n"
        header += f"date: {self.date}\n"
        header += f"category: {self.category}\n"
        header += f"tags: {self.tags}\n"
        header += f"slug: {self.slug}\n"
        header += f"author: {self.author}\n"
        header += f"summary: {self.summary}\n"

        if self.markdown:
            header += f"\n# {self.title}\n\n"
        else:
            pyperclip.copy(self.slug)

        self.location.write_text(header)
        print(f"Generated new article: {self.location}")

Once again, this is all code that I can already start to interact with, but let me add the rest of the helper functions.

In [20]:

def get_bool(subject):
    value = input(f"{subject.capitalize():>10} [y]/n? ")
    if not value:
        return True
    return False


def get_input(subject):
    value = input(f"{subject:>10}: ")
    if not value:
        print(f"No subject provided!")
        exit

    return value.strip()


def get_tags():
    tags = []
    while True:
        tag = input("       tag: ")
        if tag:
            tags.append(tag.strip().lower())
        else:
            break

    return ", ".join(tags)

You can add functions in separate cells or all together like I did here, it really doesn't matter. The one thing that you have to keep in mind though is that these cell blocks can all be executed in any order, so it's easy to have "old" code in memory that can still be used when working on other parts. In cases like that where I've changed a lot of code, I like to just Restart the Kernel and Rerun All Cells.. from the Run menu command from the JupyterLab interface.

Playing around with the code¶

So the core of the script is now loaded. Let's see what we can do. As you can tell from looking at the class, we have to instantiate an instance by providing the following:

title: str, A title for the new blog post
category: str, A category to file it under
tags: List[str], A list of strings for the tags
summary: str, A summary of of what the blog is going to be about
markdown: bool, True or False if this is going to be a markdown file, if it's not it'll set it up for a notebook

This is where the helper functions come in handy. I have the following execute when the script is ran: [on/off]

In [23]:

def main():
    title = get_input("title")
    category = get_input("category")
    tags = get_tags()
    summary = get_input("summary")
    markdown = get_bool("markdown")
    post = Article(title, category, tags, summary, markdown)
#     post.create_file()

I've commented out the last line that actually creates the file because I really don't want to create it at the moment, but let's the see the rest in action:

In [24]:

main()

Since the default is to create a markdown file, no answer was needed for the last question.

Exploring the Article object¶

Ok, so what did that actually do? Let's see!

In [25]:

post

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-25-9414b608df81> in <module>
----> 1 post

NameError: name 'post' is not defined

Oh no, what happened? Well when I normally run the script, it automatically creates the new file for me. In this instance, it created the new post but it was lost in the main function because I didn't save its state! I'll leave it up to you to explore what it contains, but it's pretty easy to see that it just filled in everything that I need to boilerplate a new pelican blog post.

How can you do the same¶

Make sure that you're working in your virtual environment and go into the directory where you have your blog project. Create a plugins directory if you don't already have one and go into it. Then clone the plugin's repos directly into it.

# from inside your project folder
mkdir plugins, cd $_
git clone https://github.com/danielfrg/pelican-ipynb ipynb

Open up your pelicanconf.py file and make the following changes/additions:

MARKUP = ("md", "ipynb")
PLUGIN_PATHS = ["./plugins"]
PLUGINS = ["ipynb.markup"]
IGNORE_FILES = [".ipynb_checkpoints"]
IPYNB_USE_METACELL = False  # set to True if you want to embedd metadata into notebook
IPYNB_SKIP_CSS = True

Now, if you want to do what I'm doing and keep the metadata in its own file and out of the notebook, you'll have to create a file with the exact same name as the notebook but with the .nbdata extension. The one for this article looks like this:

title: Playing around with Jupyter notebooks
date: 2020-03-09 19:43
modified: 2020-03-10 08:17
category: Blog
tags: jupyter, notebook, python
slug: playing-around-with-jupyter-notebooks
author: Martin Uribe
summary: Just giving the notebook plugin a test run to see how things render

Make sure there is an empty line at the end of the file. To embed directly into the notebook, modify it like this and put it in the first cell:

- title: Playing around with Jupyter notebooks
- date: 2020-03-09 19:43
- modified: 2020-03-10 08:17
- category: Blog
- tags: jupyter, notebook, python
- slug: playing-around-with-jupyter-notebooks
- author: Martin Uribe
- summary: Just giving the notebook plugin a test run to see how things render

Making it look pretty¶

Originally when the notebook was being converted into html, it was putting anchor tags at the end of headings that I didn't like it. Also, each code cell was getting a In[1] placed in front of it, indicating in which order it was run. I don't need or want any of those, any code that I post will be expected to be ran in order. To get rid of them, I created these custom CSS entries into the style sheet of my theme:

// jupyter notebook
a.anchor-link {
    visibility: hidden;
}

.prompt, .output_prompt {
    visibility: hidden;
}

Sample chart¶

I can't showcase a notebook without throwing in a bonus chart! It's nothing fancy, but wanted to illustrate what is possible to do in these notebooks. I snagged this example from the excellent book "Data Science from Scratch, Second Edition, by Joel Grus (O'Reilly). (C) 2019 Jeol Grus, 978-1-149-04113-9"

In [9]:

import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline

variance     = [1, 2, 4, 8, 16, 32, 64, 128, 256]
bias_squared = [256, 128, 64, 32, 16, 8, 4, 2, 1]
total_error  = [x + y for x, y in zip(variance, bias_squared)]
xs = [i for i, _ in enumerate(variance)]

# We can make multiple calls to plt.plot
# to show multiple series on the same chart
plt.plot(xs, variance,     'g-',  label='variance')    # green solid line
plt.plot(xs, bias_squared, 'r-.', label='bias^2')      # red dot-dashed line
plt.plot(xs, total_error,  'b:',  label='total error') # blue dotted line

# Because we've assigned labels to each series,
# we can get a legend for free (loc=9 means "top center")
plt.legend(loc=9)
plt.xlabel("model complexity")
plt.xticks([])
plt.title("The Bias-Variance Tradeoff")
plt.show()

Conclusion¶

So far, I can't complain. This interface does seem a little nicer for some reason. Perhaps I need to change the black theme of my VSCode back to normal and see if that makes a difference in the "feels" department. The notebook definitely gives me more options, but also takes some away. For instance, I can't use the liquid_tags Pelican plugin!

Like all things, it's going to take some more experimentation to see what works and what doesn't.

MongoDB CLI Intro

2020-03-07T12:27:00-06:00

Why another article on MongoDB

After publishing my article on How to install MongoDB on Linux I realized that I should have covered how to access it through the command line! Didn't occur to me that anyone reading it might not want to download and install Robo 3T!

Sorry about that!

Intro into the mongo cli app

It's simple enough. Now I'm no expert but this is what I discovered just playing around with it.

(base) ➜  ~ mongo
MongoDB shell version v4.2.3
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("39eaf67e-7de5-4874-ba3e-a6673080cfbc") }
MongoDB server version: 4.2.3
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
    http://docs.mongodb.org/
Questions? Try the support group
    http://groups.google.com/group/mongodb-user
Server has startup warnings: 
2020-03-06T20:35:19.314-0600 I  STORAGE  [initandlisten] 
2020-03-06T20:35:19.314-0600 I  STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-03-06T20:35:19.314-0600 I  STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2020-03-06T20:35:22.538-0600 I  CONTROL  [initandlisten] 
2020-03-06T20:35:22.539-0600 I  CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2020-03-06T20:35:22.539-0600 I  CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2020-03-06T20:35:22.539-0600 I  CONTROL  [initandlisten] 
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).

The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.

To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---

>

As you can tell, I haven't secured the database yet. Let's continue exploring though and see what the help command will show us.

> help
    db.help()                    help on db methods
    db.mycoll.help()             help on collection methods
    sh.help()                    sharding helpers
    rs.help()                    replica set helpers
    help admin                   administrative help
    help connect                 connecting to a db help
    help keys                    key shortcuts
    help misc                    misc things to know
    help mr                      mapreduce

    show dbs                     show database names
    show collections             show collections in current database
    show users                   show users in current database
    show profile                 show most recent system.profile entries with time >= 1ms
    show logs                    show the accessible logger names
    show log [name]              prints out the last segment of log in memory, 'global' is default
    use <db_name>                set current database
    db.foo.find()                list objects in collection foo
    db.foo.find( { a : 1 } )     list objects in foo where a == 1
    it                           result of the last line evaluated; use to further iterate
    DBQuery.shellBatchSize = x   set default number of items to display on shell
    exit
>

Ok, looks simple enough. That show dbs sounds interesting.

> show dbs
admin      0.000GB
config     0.000GB
local      0.000GB
snake_bnb  0.000GB
>

Ok, I see the snake_bnb database that we created during the MongoDB Quickstart tutorial. Let's see what's in it with that use <db_name> command, followed by show collections.

> use snake_bnb
switched to db snake_bnb
> show collections
owners
>

Now if I reference the help command again, I see that I can use db.foo.find() to do a dump of everything in it.

> db.owners.find()
{ "_id" : ObjectId("5e6243ca45ed06e7f09781fc"), "registered_date" : ISODate("2020-03-06T06:36:26.108Z"), "name" : "clamytoe", "email" : "[email protected]", "snake_ids" : [ ], "cage_ids" : [ ] }
>

If I had more entries, it looks like I could search for specific entries with db.foo.find( { a : 1 } ). I only have one record, so I should just get the same result as the previous command.

> db.owners.find( {"name": "clamytoe"} )
{ "_id" : ObjectId("5e6243ca45ed06e7f09781fc"), "registered_date" : ISODate("2020-03-06T06:36:26.108Z"), "name" : "clamytoe", "email" : "[email protected]", "snake_ids" : [ ], "cage_ids" : [ ] }
>

To exit, we just use the exit command.

> exit
bye
(base) ➜  ~

Conclusion

There you have it. A quick intro into using the mongo cli app. Simple enough for quick exploration of your databases. I'll probably be spending more time in Robo 3T though. Not only can your explore the databases through a GUI, but you can also easily add, delete, and update and entries with it.

How to install MongoDB on Linux

2020-03-06T12:53:00-06:00

Learning about MongoDB

So the other day I started going through Michael Kennedy's free MongoDB Quickstart course. Although he covered the Python code portions, he left getting everything setup, up to you. If you're on a Linux machine that uses APT, then keep on reading.

Getting MongoDB

I could have gone over to MongoDB's download page, and selected the correct version for my setup, but I wanted to stick with the package manager on my machine. The latest version as of this writing was 4.2. The first thing I had to do was find their public key and import it to APT.

Importing MongoDB's security key

To make sure that I had the required gnupg package installed, I ran the following command: sudo apt-get install gnupg. Once that was taken care off, I imported the key:

wget -qO - https://www.mongodb.org/static/pgp/server-4.2.asc | sudo apt-key add -

Creating a new source for APT

The next thing I did was to find the codename for my system's release. I did that with the following command: grep -i /etc/os-release

VERSION_CODENAME=tricia
UBUNTU_CODENAME=bionic

There it was. My Linux Mint system is based on bionic. Now to create an entry for MongoDB in APT's source list directory /etc/apt/sources.list.d/, I created mongodb-org.4.2.list with the following content:

deb [arch=amd64] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.2 multiverse

I had to look at other entries in there until I got the format correct. Don't worry, if you get it wrong, it will let you know when you try to do an update!

Update APT

At this point it was time to update APT and install mongoDB:

sudo apt-get update
sudo apt-get install -y mongodb-org

Starting MongoDB

Once everything was installed, it was time to fire up the service so that I could connect to it.

sudo systemctl start mongod

Now, that command will start the server for this session only. Which means that the next time that I reboot, I will have to start it again. I wanted the service to always be available, so I did the following:

sudo systemctl enable mongod.service
sudo systemctl daemon-reload
sudo systemctl restart mongod.service

To verify that it was running, I used this command: systemctl status mongod

● mongod.service - MongoDB Database Server
   Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: e
   Active: active (running) since Fri 2020-03-06 20:35:16 CST; 12s ago
     Docs: https://docs.mongodb.org/manual
 Main PID: 8569 (mongod)
   CGroup: /system.slice/mongod.service
           └─8569 /usr/bin/mongod --config /etc/mongod.conf

Mar 06 20:35:16 SERENiTY systemd[1]: Started MongoDB Database Server.

If you want the server to restart after a crash, you would need to add Restart=always under the [Service] section of /lib/systemd/system/mongod.service. I didn't add that.

At this point, a reboot would be useful to make sure that it starts up at boot.

If you decide that you don't want it running all of the time, just disable it.

sudo systemctl stop mongod
sudo systemctl disable mongod

Installing Robo 3T

Michael recommended Robo 3t so I decided to give it a try. I went to their download page and selected the linux version. My connection at home is terrible, so I try not to download anything through the browser. I copied the link and used wget instead.

cd ~/Downloads
wget -c https://download-test.robomongo.org/linux/robo3t-1.3.1-linux-x86_64-7419c406.tar.gz

Once it had finished downloading, I extracted the file to my home directory:

mkdir ~/robo3t
tar -xvzf robo3t-1.3.1-linux-x86_64-7419c406.tar.gz -C ~/robo3t

Adding a custom menu entry

I like to add all of my programs to the "start" menu, so I then did the following:

cd ~/robo3t
wget https://robomongo.org/static/robomongo-128x128-129df2f1.png -O robomongo.png
ln -s robo3t-1.3.1-linux-x86_64-7419c406/bin/robo3t robo3t

Right-click the Mint start button
Click on Configure...
Click on the Menu button from the dialog window and then click on Open the menu editor
From the Main Menu widget, click on the Programming section and then click on the New Item button
- Fill in the Launcher Properties.
- Select the robomongo.png image that we downloaded previously.
- Give it a Name, Robo 3T.
- For the Command select the symlink that we created previously.
- Enter whatever you want under Comment and click OK.
Close all of the open dialogs.
Admire you're awesomeness

Before you can start using Robo 3T, you will have to get to the point of the tutorial where you add a user.

Once you've made an entry, go ahead and start it Robo 3T. You will be presented with a MongoDB Connections dialog. Click on the Create link to open up the Connection Settings dialog.

For the Name enter snake_bnb, the name of the database that was created in the tutorial and Save.

Click on the Connect button to access it.

Conclusion

Quite a few steps to get here, but now I am setup and ready to continue on with the course and hopefully you will be too!

Playing around with adding plugins to Pelican

2020-03-03T13:20:00-06:00

Some parts of my blog were looking pretty blah

Math syntax

I've been tweaking my blog like crazy. Just making minor changes here and there. I wanted to make things look a little better.

For instance, for my rescaling article, I had the following mathematical syntax:

ScaledQuantity = (ActualQuantity-MinQuantity)/(MaxQuantity-MinQuantity)

What the hell is that? That looks horrible! I know that markdown supports Latex, so I looked and found this awesome resource: Writing Mathematic Formulars in Markdown

ScaledQuantity = \frac{ActualQuantity-MinQuantity}{MaxQuantity-MinQuantity}

Um, not quite there yet. Something was up. I searched to see if Pelican had any plugins for rendering math and sure enough, found that pelican-plugins included render_math. I followed the instructions and cloned the repo into my blog directory. Probably should have dropped it somewhere outside of it, but I added it to my .gitignore and it's fine...

I added the following to my pelicanconf.py file:

PLUGIN_PATHS = ['./pelican-plugins']
PLUGINS = ['render_math']

I tried the previous code once again and was still getting the same results. After some more reading I found that you have to enclose the mathematical equation within double $.

$$ScaledQuantity = \frac{ActualQuantity-MinQuantity}{MaxQuantity-MinQuantity}$$

NICE!

Markdown tables

My original tables were so bland that it was almost hard to tell that they were even tables! Unfortunately I didn't save a screenshot of it, so I can't show you what they looked like, but I can definitely show you the end result! Thanks to mockaroo I was able to generate some mock data relatively quickly.

id	first_name	last_name	email	gender	ip_address
1	Odilia	O'Meara	[email protected]	Female	244.53.86.185
2	Marcelo	Tunna	[email protected]	Male	128.140.17.56
3	Farris	Roston	[email protected]	Male	111.104.78.209
4	Gene	Benzies	[email protected]	Male	67.169.248.21
5	Ellsworth	Goodered	[email protected]	Male	227.48.46.133
6	Jan	Garnsworth	[email protected]	Male	138.13.211.142
7	Robby	Rosenstengel	[email protected]	Female	105.76.137.198
8	Granthem	Rymer	[email protected]	Male	15.60.66.25
9	Claire	Stuther	[email protected]	Female	5.36.242.109
10	Maggy	Jearum	[email protected]	Female	179.12.188.75

CSS changes

I hope it doesn't bite me in the butt later on, but I'm forcing the table headers into uppercase. In case anyone is curious, this are the styles that I added to the theme:

table {
  border-collapse: collapse;
  margin-left: auto;
  margin-right: auto;
  width: 90%;
}

td, th {
  border: 1px solid #ddd;
  padding: 8px;
}

tr:nth-child(even){background-color: #f2f2f2;}

tr:hover {background-color: #ddd;}

th {
  padding-top: 12px;
  padding-bottom: 12px;
  text-transform: uppercase;
  font-weight: bold;
  background-color: #607D8B;
  color: white;
}

.math {
  font-size: medium;
}

The default math syntax rendering was kind of small, so I changed the font-size to medium.

On a side note

I noticed that some of my changes wouldn't take effect until after I removed the pycache and .pytest_cache directories. In order to automate this, I modified my Makefile's clean code to the following:

clean:
    [ ! -d $(OUTPUTDIR) ] || rm -rf $(OUTPUTDIR)
    rm -rf __pycache__
    rm -rf .pytest_cache

Conclusion

This web development stuff is pretty complex. There are lots of moving parts, so I give all of my fellow web developers a high-five for being so awesome! I can see how this can be pretty overwhelming if you're new to this, but just keep at it and you'll get there.

This old dog can still learn some new tricks!

Finally getting the hang of working with Pelican!

2020-02-28T09:50:00-06:00

Stumbling into a nice Pelican workflow

If you've seen my source repo for this site, you know that I followed Erik O'Shaughnessy's Run your blog on GitHub Pages with Python tutorial. It's a great tutorial, I specifically love his one weird trick, but it leaves so many unanswered questions. Go read it now if you haven't and we'll pick up where he leaves off.

Luckily he does provide the link to the official documentation. It was through that link that I found Pelican's tutorials repo. Specifically Matt Makai's introductory tutorial. I'll admit, that I only skimmed over it, it's kind of long. I was looking for anything that would jump out at me. What caught my eye was how simple it all was to automate things with make!

I took a quick look at mine and saw what commands were defined.

Makefile for a pelican Web site

Usage:
   make html                           (re)generate the web site
   make clean                          remove the generated files
   make regenerate                     regenerate files upon modification
   make publish                        generate using production settings
   make serve [PORT=8000]              serve site at http://localhost:8000
   make serve-global [SERVER=0.0.0.0]  serve (as root) to :80
   make devserver [PORT=8000]          serve and regenerate together
   make ssh_upload                     upload the web site via SSH
   make rsync_upload                   upload the web site via rsync+ssh  
   make github                         upload the web site via gh-pages

Set the DEBUG variable to 1 to enable debugging, e.g. make DEBUG=1 html
Set the RELATIVE variable to 1 to enable relative urls

Looking at the Makefile source is where I noticed that for local development, it just uses the pelicanconf.py script, but for publishing it uses publishconf.py.

Makefile[Lines 8-9]

CONFFILE=$(BASEDIR)/pelicanconf.py
PUBLISHCONF=$(BASEDIR)/publishconf.py

Looking at publishconf.py I saw that it actually imports pelicanconf.py and then overrides some of the variables.

publishconf.py[Lines 12-19]

from pelicanconf import *

# If your site is available via HTTPS, make sure SITEURL begins with https://
SITEURL = "https://clamytoe.github.io"
RELATIVE_URLS = False

FEED_ALL_ATOM = "feeds/all.atom.xml"
CATEGORY_FEED_ATOM = "feeds/{slug}.atom.xml"

AHA! This is were I had a eurika moment!

I had noticed a problem with my site. Whenever I clicked on the header link, instead of taking me back to the home page it would just reload whatever page I was on. I was able to get around that by just specifying the SITE_URL variable in pelicanconf.py. Unfortunately, that broke testing the blog locally because all the links would point to the live site on GitHub!

What I'm trying to say here is that in I was able to resolve both of these issues by making the following changes.

pelicanconf.py[Lines 16-16]

SITEURL = ""

publishconf.py[Lines 12-12]

from pelicanconf import *

Another issue that I was having was that I couldn't figure out how to get my favicon.ico to be treated as a static file and automatically dumped into the output directory. My solution had been to write a little bash script to copy it over for me, but that was kind of tedious. But now that I was going to use the Makefile I decided to automate it in there!

I added the following lines to it:

Makefile[Lines 10-10]

FAVICON=$(INPUTDIR)/static/favicon.ico

Then after the html, regenerate, and publish commands, I added:

Makefile[Lines 44-46]

html:
	$(PELICAN) $(INPUTDIR) -o $(OUTPUTDIR) -s $(CONFFILE) $(PELICANOPTS)
	cp $(FAVICON) $(OUTPUTDIR)

Makefile[Lines 53-55]

regenerate:
	$(PELICAN) -r $(INPUTDIR) -o $(OUTPUTDIR) -s $(CONFFILE) $(PELICANOPTS)
	cp $(FAVICON) $(OUTPUTDIR)

Makefile[Lines 79-81]

publish:
	$(PELICAN) $(INPUTDIR) -o $(OUTPUTDIR) -s $(PUBLISHCONF) $(PELICANOPTS)
	cp $(FAVICON) $(OUTPUTDIR)

NOTE: I had initially added that to all of the commands, but the ones that run the server, don't actually get to that step.

When I Ctrl+C out of it, it just exists...

New workflow

With that all set, things are now much easier and make more sense. What I would recommend, is to run the make clean command and commit it to your content branch. Now you will have a clean slate to start with.

You can now add any new content and do the following commands:

git add .
git commit -m "Your description..."
git push origin content

NOTE: Of course, only push your files when you're ready to, but you get the idea.

Now you're ready to generate the actual site, here's how that's now done.

Generate the site

If you just want to generate the site this is the command that you want:

make html

Then you can either run the Pelican server or the Python one.

Pelican server

pelican -r --listen

Python server

cd output
python -m http.server

Both of those will make the site accessible at http://localhost:8000

Generate site and preview

Now, if you know that you will be wanting to preview your site, a much better way is to use this command:

make devserver

Not only will this generate the site, but will also start the Pelican server automagically for you! This option comes in handy when you are tweaking your site's theme. I wanted to change the colors on mine and doing it like this automatically detected the changes and served them up.

Publish the site

This command generates the site using the publishconf.py script, so if you specified your site's url in there, it will add it to the generated html pages. It's nice for making sure all is working correctly and enabling your Google Analytics; if you're into that sort of thing.

make publish

Publish and push to github

Now, we're cooking with Crisco! This command will generate the final site using publishconf.py, merge the output into your master branch, and push it up the hubs; all in one step!

make github

Cleaning up

Now that you've pushed to the hubs, removed the generated files:

make clean

If you run git status, you should see that there are no changes and that you are ready to add more content. Nice!

Conclusion

This was a great learning experience for me. Not only did I update the color scheme of my site, I also fixed some annoying issues that I was having. Now I don't have to look up any ghp-import ... command and things just got a heck of a lot more pleasant.

In summary:

touch content/posts/new-blog-post.md
vi $_
# write the blog content
git status
git add .
git commit -m "New blog post."
git push origin content
make github
make clean

On a side note

I was also having a weird problem with my search feature. I had to remove the machine code from my cryptography blog post because it was completely screwing up the JavaScript. Once I got that working, I then noticed that whenever I tried to do a search from any of the category pages, that it was looking for /category/search.html. That page doesn't exists, so I was baffled as to why that was going on. I took a look at the pelican-mg theme's Jinja templates and found the issue in base.html. I simply had to add {{ SITEURL }} in the action part of the form:

<form class="uk-search" action="{{ SITEURL }}/search.html" data-uk-search>

Now it works beautifully!

Of course, with me being an upstanding netcitizen, I submitted a pull request to fix the issue :).

Pytest fixtures with tear down

2020-02-19T17:40:00-06:00

Setting up pytest fixtures with teardown code

Today after listening to Brian Okken's Test & Code podcast with Anthony Shaw, I was reminded that I wanted to install Anthony's Python Security Plugin for JetBrain's PyCharm IDE. That was a breeze to install and it just worked out of the box. I fired it up and it automatically checked my code and called it good.

I then started to write some tests and while checking a password, it alerted me to an issue. It made a suggestion to fix it, so I accepted and it updated my code and even added the proper import statement!

FROM:

    assert dummy.password == "admin"

TO:

from secrets import compare_digest

import pytest

from passlock.entry import Entry


@pytest.fixture(scope="module")
def dummy():
    return Entry("router", "http://192.168.2.1", "admin", "admin")


def test_entry(dummy):
    assert dummy.name == "router"
    assert dummy.url == "http://192.168.2.1"
    assert dummy.username == "admin"
    assert compare_digest(dummy.password, "admin")

Although, this is just a dummy password for testing, it doesn't hurt to build up some good habits!

On to the fixture tear down

This is when I got to the part where I had to test the creation of my private and public keys. Since I was creating them for testing and didn't want them around afterwards, I decided that I needed to make them go away after the tests were done.

I was already using fixtures so that I didn't have to recreate the instances of my classes for each test; generating the keys take a bit of time! I went crazy an opted to make them future proof with 4096-bits...

The challenge was figuring out how to tear them down. I did some initial searches and was coming up blank. I then looked up the pytest guru, Brian's blog about the topic, but his examples were too simplistic for what I was trying to do. All they demonstrated was how to print a message, it didn't actually show how to return anything, and then have it torn down afterwards.

Brian's example:

from __future__ import print_function
import pytest

@pytest.fixture(scope='module')
def resource_a_setup(request):
    print('\nresources_a_setup()')
    def resource_a_teardown():
        print('\nresources_a_teardown()')
    request.addfinalizer(resource_a_teardown)

def test_1_that_needs_resource_a(resource_a_setup):
    print('test_1_that_needs_resource_a()')

def test_2_that_does_not():
    print('\ntest_2_that_does_not()')

def test_3_that_does(resource_a_setup):
    print('\ntest_3_that_does()')

I ended up reading the actual pytest documentation; who would have thunk?! I'll do you guys a solid and cut out the noise and show you what I did:

import pytest


@pytest.fixture(scope="module")
def custom_locker():
    v = Vault(".toe", "tlock")
    yield v
    v.plock.unlink()
    v.pub.unlink()
    v.base.rmdir()


def test_vault_custom(custom_locker):
    assert custom_locker.loc == ".toe"
    assert custom_locker.key_name == "tlock"

Setting the scope of the fixture to module makes it so that it doesn't actually do the tearing down until after all of the tests are completed. Perfect!

The variables in the Vault instance are Path objects, so it was simple enough to just unlink() the files and then rmdir() the directory after it's been emptied. Of course, I tried removing the directory first, but it wasn't having any of that until it was empty...

Into the cryptography rabbit hole

2020-02-14T07:00:00-06:00

Venture Into Cryptography with PyCryptodome

Learning about how to encrypt files

As dumb as it is, I have an unsecured text file that I use to keep track of all my username and passwords for all of the sites that I frequent. In order to not be so "hackable", I've decided to encrypt the file.

I had recently written a couple of coding challenges for Pybites Code Challenges site, using the cryptography module. One was a coding challenge and the other a testing one. Unfortunately, that module is NOT part of Python's Standard Library!

I didn't specifically install it, but it seems to come pre-installed with Anaconda, which I use. I was looking through all of the modules that I had, when I came across the cryptography one. That's when I first got the idea for the challenges. It started as a single challenge, but I really liked how the tests came out and decided to split it up into coding and testing bites.

I pushed them to the Pybites Github private repo and it sat there for a bit before Bob Belderbos decided to add them. That's when the problems started! He discovered that the module was not available so he installed it. It worked perfectly for the coding challenge, but for the testing one, when the call to the Pytest/Mutpy instance is made, it looses the path to the module!

We were discussing the issue on the Pybites Slack channel when Mike Driscoll the creator of The Mouse vs Python, mentioned the PyCryptodome module. He initially thought that the problem was with installing the cryptography module. In the end, Bob was not able to figure out how to get it to work, so the challenge had to be removed from the platform.

Regardless, I was intrigued by pycryptodome, so I started to look into how it worked. It seemed pretty cool, so that's when I came up with the idea of encrypting my password file. It took a little bit of work, but I was able to encrypt the file into a binary. The problem came when I started to try and retrieve the data. The module complained about the data being too large!

I decided to create an encrypted binary for each entry. So far that's worked, but now I have to figure out how to scan the directory and list the ones that are available in order to be able to retrieve/update them. I've been using the logging module as well, so now I have to figure out how to do some log rotation to keep from taking up too much drive space. I might even play around with creating a UI for it, to make it easier to use.

Since I used my toepack Cookiecutter template, it's a package. It's gotten big enough that I've moved the two classes, (Entry, Vault) into their own files. It works out pretty good because the app.py file now only contains:

from .entry import Entry
from .vault import Vault, logger


def main():
    logger.debug("Entering main.")
    ip = Entry("ip", "http://192.168.2.1", "admin", "admin")

    v = Vault()
    v.encrypt_file(ip)
    v.decrypt_file(ip.name)


if __name__ == "__main__":
    logger.debug("Running as a module.")
    main()

Separating the code like this will come in handy when it comes to either creating a Click CLI interface or a GUI one.

The log file for that run looks like this:

2020-02-13 14:05:17,812 - [INFO] - root - Creating vault: /home/mohh/Documents/.passlock/ip.bin
2020-02-13 14:05:17,812 - [INFO] - root - Importing public key: plock.pub
2020-02-13 14:05:17,812 - [INFO] - root - Public key accessed: /home/mohh/.passlock/plock.pub
2020-02-13 14:05:17,815 - [INFO] - root - Generating random byte for session key
2020-02-13 14:05:17,815 - [INFO] - root - Generating encrypted session key
2020-02-13 14:05:17,819 - [INFO] - root - Generating AES cipher
2020-02-13 14:05:17,820 - [INFO] - root - Encrypting data to create cipher text
2020-02-13 14:05:17,820 - [INFO] - root - Writing cipher text to: ip.bin
2020-02-13 14:05:17,820 - [WARNING] - root - /home/mohh/Documents/.passlock/ip.bin already exists!
2020-02-13 14:05:17,820 - [INFO] - root - Successfully wrote to: /home/mohh/Documents/.passlock/ip.bin
2020-02-13 14:05:17,821 - [INFO] - root - Opening file for reading: /home/mohh/Documents/.passlock/ip.bin
2020-02-13 14:05:17,821 - [INFO] - root - Importing private key: plock
2020-02-13 14:05:17,821 - [INFO] - root - Private key accessed: /home/mohh/.passlock/plock
2020-02-13 14:05:18,150 - [INFO] - root - Extracting encrypted session key
2020-02-13 14:05:18,151 - [INFO] - root - Generating RSA cipher key from imported private key
2020-02-13 14:05:18,151 - [INFO] - root - Decrypting extracted session key
2020-02-13 14:05:18,171 - [INFO] - root - Session key was successfully restored
2020-02-13 14:05:18,171 - [INFO] - root - Generating AES cipher from session key
2020-02-13 14:05:18,172 - [INFO] - root - Decrypting and verifying cipher text
2020-02-13 14:05:18,172 - [INFO] - root - Closing: ip.bin
2020-02-13 14:05:18,172 - [INFO] - root - Generating new Entry object from decrypted data
2020-02-13 14:05:18,172 - [INFO] - root - Returning decrypted text for: ip

Not sure if that's too much information but logging each step in the process really comes in handy when trying to figure out issues at work. I've tried to keep all sensitive data out of it unless the logging is switched to debug.

Retrieving the encrypted data looks like this:

Successfully created: /home/mohh/Documents/.passlock/ip.bin

[ip]
INFO: http://192.168.2.1
USER: admin
PASS: admin

NOTE: These are not real username and passwords!

Rescaling values for DataScience

2020-02-13T08:00:00-06:00

Scaling values

Today while reading Data Science Algorithms in a Week, from packt, I came across the concept of rescaling values so that when measuring their distances they would be more relevant. The dataset consisted of "House Ownership":

Age	Annual income in USD	House ownership status
23	50,000	Non-owner
37	34,000	Non-owner
20	100,000	Owner
35	130,000	Owner

etc..

The aim being to predict whether a person that is 50 years old with an income of $80,000, would own a home so that he could be targeted for home insurance.

k-Nearest Neighbor's are currently being covered and for this exercise a k = 1 is to be used. Using either a Euclidean or Manhattan distance wouldn't matter because the distances between these values are too great. In comes rescaling!

The formula use is:

$$ScaledQuantity = \frac{ActualQuantity-MinQuantity}{MaxQuantity-MinQuantity}$$

So for this dataset, both the Age and the Annual income wound have to be adjusted.

$$ScaledAge = \frac{ActualAge-MinAge}{MaxAge-MinAge}$$

$$ScaledIncome = \frac{ActualIncome-MinIncome}{MaxIncome-MinIncome}$$

After scaling, the adjusted dataset would look something like this:

Age	Scaled age	Annual income in USD	Scaled annual income	House ownership status
23	09375	50,000	2	Non-owner
37	53125	34,000	4	Non-owner
20	0	100,000	7	Owner
35	46875	130,000	1	Owner
50	9375	80,000	5	?

Now a 1-NN algorithm with a Euclidean metric could easily be used to find out if the last person is more than likely to own a home. Without the rescaling, the algorithm would have yielded different results.

Keeping it short today, but hopefully it was a helpful tip!

This is my first post here

2020-02-12T12:00:00-06:00

Getting my feet wet with this blogging stuff

It's not much to look at, but my goal is to show what I'm learning as I go. There's no real "topic" but it will mostly be about what I'm working on and what I have to learn to get it done. Hope that it's of interest to someone...

Ramblings

More advanced Dataclass sorting

More advanced Dataclass sorting

Pretty up the output

Using a sort_index

Dataclasses field(init=False)

Dataclasses field(repr=False)

Dataclasses frozen

Setting class variables with setattr

Conclusion

Troubleshooting Connectivity Issues

Troubleshooting Connectivity Issues

Some Background

Attacking the Problem

The Problem

Firewall: Custom Rule

Firewall: Program Rule

Conclusion

Benchmarking Python Code

Need to compare which code is faster, read on to find out how

Move Zeroes

Decorator: ID Checker

Decorator: Timer

Decorating the Functions

Generating the Numbers

Test Function

Benchmarking

A Better Way to Benchmark Code

Summary

Update

Advanced type annotations

How to add type annotations to Python dataclasses

Sample program

Hell week

What I ended up with

Learning points

The class definitions

__post_init__

Volume property

Inheriting the main class

Property init values

Overwriting methods

Conclusion

Cleaning up Anaconda environment.yml files

Recreating virtual environments with Anaconda

Table of Contents

What is an environment.yml file

How do you use an environment.yml file

How do you create and environment.yml file

How to prepare an environment.yml file for mass use

How to use the cleanup-yml.py script

How to create an environment.yml without version lock in

Conclusion

Bash script into a Makefile

Using make can literally make your life easier

What is a make and what is a Makefile

The bash script

Creating the Makefile

Variables

Referencing variables

Shell commands

Functions / macros

Using the macro

Shared flags

Phony rule

Default rule

Rules

The debug rule

The clean rule

The help rule

Conclusion

Testing ABC's with abstract methods with Pytest

So you want to test ABC's with pytest

How to test abstract methods

Conclusion

Playing around with Jupyter notebooks

Seeing how notebooks compare to just plain Markdown¶

What is a Jupyter Notebook¶

Update to my newpost script¶

Imports and constants¶