1. What is an exception?
2. What is an exception chaining?
3. Why is it important to use it?
4. How to chain exceptions in Python?

Let’s get started!

Introduction

Exception is a software error we, software developers, can create and handle (or not). It breaks the normal flow of the software and let us the option to run a special code when the software encounters one.

For example, if we divide a number by zero, we get an exception:

>>> 1/0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero

In this case, ZeroDivisionError is the exception. If we want to handle this exception we can wrap the dangerous code with try/except statements:

>>> try:
...     1/0
... except ZeroDivisionError:
...     print("Cannot divide by zero")
... 
Cannot divide by zero

What happens if an exception is thrown while handling another one?

>>> try:     
...     1/0
... except ZeroDivisionError as e:
...     print("error:", invalid_keyword_argument="oops")
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
TypeError: 'invalid_keyword_argument ' is an invalid keyword argument for print()

It’s nice to see that the interpreter knows that an exception occurred while handling another one.

Exception chaining

This is a programming technique which we can use by throwing an exception while catching one and link them together. In Python, there is an elegant way for doing so. We can achieve it by using raise ... from ...:

>>> class MyException(Exception): pass
... 
>>> try:
...     1/0
... except ZeroDivisionError as e:
...     print("Cannot divide by zero")
...     raise MyException("Failed to calculate the math formula") from e
... 
Cannot divide by zero
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
ZeroDivisionError: division by zero

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 5, in <module>
Exception: Failed to calculate the math formula

Earlier we saw the interpreter knows an exception was thrown while handling one, but this time, the message is different. Now, it tells us the exceptions are connected. If we don’t chain the exceptions together, the printed message implies that the exception-handling-code has issues regardless the original exception.

Importance

Exception chaining let us the option to add information to the original exception without discarding the original information. By doing so, each layer in our software design can:

1. add its information to the exception- every layer can describe the layer’s status/problem without referring to some low/high level layers
2. keep abstraction levels- when the levels are abstraced, we can maintain them easier than if they’re tangled/convulated.

References

https://docs.python.org/3/reference/simple_stmts.html#grammar-token-raise-stmt

https://en.wikipedia.org/wiki/Exception_chaining

If you have any questions feel free to comment or contact me directly.

Thank you for reading this. Please share and visit soon again,

Orian Zinger.

The reason for rejecting this PEP is:

A quick poll during my keynote presentation at PyCon 2007 shows this proposal has no popular support. I therefore reject it.

~ Guido van Rossum

Lots of software developers who wanted to use it searched for substitutions. They found lots of guides to mimic the behavior. Even the official documentation suggests using if ... elif ... elif ... else when we desire to use switch case.

Soon, in Python 3.10, we’ll be able to do that big time. The new feature is much more than a simple switch case.

You see, in contrast to C language, the structural pattern matching feature can use instances, guards, and more when using a case statement.

For example, this is how we use it in C language:

switch(expression) {
   case constant_expression:
      statement();
      break;
   case another_constant_expression:
      another_statement();
      break;
   default:
      default_statement();
}

The equivalent Python code looks like this:

match expression:
    case constant_expression:
        statement()
    case another_constant_expression:
        another_statement()
    case _:
        default_statement()

First of all, it looks cleaner which is great, yet not surprising because most of (all?) Python code looks cleaner than C code.

Whenever we want to match a value, the interpreter tries to match it to different cases from top to bottom and then it automatically breaks the statement. It means we don’t need to use the break statement. If we try to use a break, we get an exception:

SyntaxError: 'break' outside loop

If we want to join cases together, we can use Union.

As I wrote above, This is not a simple switch case. We’re not done yet. We can do something crazy like this (output in comments and explanation afterward):

from dataclasses import dataclass


@dataclass
class Point:
    x: int
    y: int


def match_point(point):
    match point:
        case Point(_, 0):
            print("This point is located on the X axle")
        case Point(0, _):
            print("This point is located on the Y axle")
        case Point(x, y) if x == y:
            print("This point has identical x and y values")
        case Point():
            print("This point is not special")
        case dict():
            print("Why is there a dict?")
        case int() | float():
            print("A number?!")
        case _:
            print(f"This is not a point, but a {type(point)}")


match_point(Point(x = 0, y = 42))  # This point is located on the Y axle
match_point(Point(x = 42, y = 42))  # This point has identical x and y values
match_point(Point(x = 4, y = 2))  # This point is not special
match_point(Point(x = 0, y = 0))  # This point is located on the X axle
match_point({"this is not": "a point at all!"})  # Why is there a dict?
match_point(42)  # A number?!
match_point(4.2)  # A number?!
match_point(None)  # This is not a point, but a <class 'NoneType'>

What do we have here? Main things in this code:

1. Line #5: Dataclass for holding properties of a Point class: x and y
2. Line #10: Function that matches points (and some other stuff too)
3. Line #14: We can create a generic pattern for the Point class. In this case, the code accepts every point on the y axle.
4. Line #16: We can use a guard inside the case statement. No need to add an else clause.
5. Line #18: We can create a generic pattern for the Point class. In this case, the code accepts every point.
6. Line #22: We can put a union of cases together.
7. Line #24: The generic pattern that receives every value is optional. The default behavior when no pattern is found is doing nothing.

Note that we don’t create an instance of a class when we “instantiate” the class in the case statements. We create a pattern there. The __new__ and __init__ functions aren’t called as you can see:

class A:
	def __new__(cls, *args, **kwargs):
		print("new")
		return super(A, cls).__new__(cls)

	def __init__(self):
		print("init")


a = A()  # output: new\ninit


def foo():
	match a:
		case A():  # no output
			pass


foo()

Performance comparison

How long does it take? Which way is better? The old if / elif statements chain or the new structural pattern matching feature? I added a dictionary implementation to the comparison, even though it can’t hash every element and sometimes we need to do more than returning a value.

Scenarios

1. Comparing a number to one case
2. Comparing a number to one thousand cases
3. Comparing an instance with two members to one case
4. Comparing an instance with two members to one thousand cases
5. Comparing a list with three items to one case
6. Comparing a list with three items to one thousand cases

You can see the code here. You’re welcome to fork it, test it, and push your numbers

Before I ran the tests I tried to guess the results. Since the interpreter matches a value, it knows that each comparison contains the same value. In if / elif statements, the interpreter doesn’t know in advance it’s about to compare the same value each time. Thus, the interpreter should work faster when it has lots of cases to compare to.

Comparing something to one case should be close enough because, in both situations, we compare only once.

Results

I run it using Python 3.10.0a7+ and Intel’s i7 6700 CPU.

As suspected, performance improvement is crucial when we have lots of cases. Obviously, the dictionary with the O(1) efficiency is much better than the others O(n) efficiency

Summary

It’s interesting to see that 15 years after the feature proposal, we get such a strong pattern matching mechanism.

Note that usually we don’t write a thousand cases, but the way it looks compared to if / elif, the better maintenance, and the performance improvement are great things and I can’t wait to start using it when needed.

References

https://www.tutorialspoint.com/cprogramming/switch_statement_in_c.htm

https://www.youtube.com/watch?v=-79HGfWmH_w

If you have any questions feel free to comment or contact me directly.

Thank you for reading this. Please share and visit soon again,

Orian Zinger.

Sometimes though, you can find one character right before the left apostrophe (or quotation mark). In this article, I’ll explain the difference between the prefixes you can write there.

There are four possible literals:

1. u – unicode
2. b – byte
3. r – raw
4. f – formatting

Literals comparison

Regular string

string = 'Hello world'

This one is pretty self-explanatory.

Unicode string

string = u'Hello world'

What makes our string so special? Well, we can put unicodic characters there too:

string = u'שלום עולם'

Python 3 made all strings unicodic by default, so there is no need to mark it. Actually, in Python 3.0-3.2 marking strings with u literal is considered as an error. They removed the error in 3.3 for making the code compatible with Python 2.

Since Python 2 is a dead language, I recommend not using it at all. Thus, I won’t elaborate on this one.

Byte string

not_a_string = b'Hello world'

Don’t call it a string

If it looks like a string, it’s shown in the “Python Strings Prefixes” article and printed like a string, it’s a string.

Actually, the type of this variable is bytes and not string.

What’s the difference? A variable of type bytes is a sequence of bytes. Each byte is a number in the range of 0-255.

A variable of type string is a sequence of characters. Each character is an encoded number. Since they’re different, comparing similar values returns False:

>>> 'Hello world' == b'Hello world'
False

When is it useful?

There are lots of different cases, but the one I think is the most important is when we want to send data from one socket to another:

import socket

# Server:

serverSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serverSocket.bind(('127.0.0.1', 1234))
serverSocket.listen()

clientConnection, _ = serverSocket.accept()
dataFromClient = clientConnection.recv(1024)
print('Data type:', type(dataFromClient), 'decoded data:', dataFromClient.decode())
response = b'Hello Client!'  # Looks better than 'Hello Client!'.encode()
clientConnection.send(response)

# Client:

clientSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
clientSocket.connect(('127.0.0.1', 1234))

data = b'Hello Server!'  # Looks better than 'Hello Server!'.encode()
clientSocket.send(data)
dataFromServer = clientSocket.recv(1024)
print(dataFromServer.decode())

# output:
# Data type: <class 'bytes'> decoded data: Hello Server!
# Hello Client!

This program creates a server socket and waits for a connection. A client socket sends a message to the server socket.

Take a look at line #20. We encode the string to bytes before sending it to the server because sockets understand bytes, not strings. A string is an abstraction to bytes. The recipient of the data decodes the bytes back to a regular string since it’s easier to handle a string.

Note: don’t forget to shut down the socket and close its file descriptor

Raw string

string = r'Hello\tworld'

A raw string makes (almost) all characters in the string as literals, so when we define it or use it, the interpreter won’t adjust it accordingly. In our case, \t is the way to encode in strings an ASCII Horizontal Tab (TAB). By using raw string, we tell the interpreter to keep the literals \ and t as-is. Let’s print this string with/without r prefix and compare the results:

print('hello\tworld')  # output: hello	world
print('hello\\tworld')  # output: hello\tworld
print(r'Hello\tworld')  # output: hello\tworld

When is it useful?

It’s useful mostly in these cases:

1. Regular expression: Usually, RegEx templates are hard to read and we use lots of backslashes. By declaring the template as raw, we can reduce the number of backslashes.
2. File paths in Windows: When we want to point to a file in Windows, we use something like C:\path\to\folder\file.extension. Writing two backslashes (or forgetting to use two) is redundant if we use raw strings. Note that using os.path.join function is better than building the path manually in most cases.

Formatting string

name = 'world'
string = f'Hello {name}'

In Python there are a couple of ways to format a string:

1. Using format function: 'Hello {}'.format('world')
2. Using % syntax: 'Hello %s' % 'world'
3. Using the f prefix as in the example above

When is it useful?

When we want to format a string with lots of substitutes, we have a string with lots of {} or %s and we need to remember the order or the names of the parameters when we provide them. For example:

# without parameters names:
string = 'Name: {}, age: {}, spiritual animal: {}'.format('Orian', 30, 'kitten')
# with parameters names:
string = 'Name: {name}, age: {age}, spiritual animal: {animal}'.format(name = 'Orian', age = 30, animal = 'kitten')

We can simplify the formatting procedure by declaring the string with f prefix:

my_name = 'Orian'
my_age = 30
my_spiritual_animal = 'kitten'
string = f'Name: {my_name}, age: {my_age}, spiritual animal: {my_spiritual_animal}'

Now, we don’t need to remember the order nor the names of the parameters since the interpreter evaluates the code inside the {}. This means that when we use the f prefix, we must format the string immediately. If we can’t know the values of the substitution on string declaration time, we need to use the other ways of formatting.

Note that usually, we don’t declare the variables just right before, but calling some functions or members for that.

Mixing literals

If we want to create a formatted raw string, for example, we can combine those literals:

name = 'world'
print(fr'Hello\t{name}')  # output: Hello\tworld

Note that not all literals can be combined. For example, we can’t combine f and b since a `bytes` variable cannot be formatted.

Incase-sensitive

Those literals are incase-sensitive. You can use lower letters, capital letters, a mix of cases. I don’t recommend using mixed cases though. It looks weird.

References

https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals

https://pythontic.com/modules/socket/send

https://stackoverflow.com/questions/2464959/whats-the-u-prefix-in-a-python-string

https://stackoverflow.com/questions/2081640/what-exactly-do-u-and-r-string-flags-do-and-what-are-raw-string-literals

If you have any questions feel free to comment or contact me directly.

Thank you for reading this. Please share and visit soon again,

Orian Zinger.

Let’s divide the project to two parts:

1. Taking a photo
2. Uploading the photo to Google Drive

Say Cheese

Since we want to save costs, we would like to use only one camera for capturing the food. Thus, we need a wide field of view. We can achieve it by:

1. Using a fisheye lens
2. Increasing the distance between the camera and the food

Those are the reasons for me to buy an ultra-wide 170° lens and install it on the fridge’s door. This worked perfectly… in theory.

The software doesn’t know when the door is open with “the right angle” for capturing the food. Thus, it keeps taking photos indefinitely (every couple of milliseconds). When the door is closed, there is no light in the fridge, so we don’t want to save such a photo. How can we know without installing more hardware if the door is closed or not?

I wrote a code the checks the photo. If it’s too dark, it takes another photo without saving the dark one.

The camera I bought supports automatic exposure. It means that when the photo it too dark, it increases automatically the exposure time, so the photo will be brighter. When we open the door, there is a lot of light that’s being captured by the camera which makes the photo very bright.

I solved this issue in a similar approach to the previous, too dark photos, issue. If the photo is too bright, it takes another one without saving the bright one.

Uploading

There is a Python module for Google Drive which is great. It needs the credentials for creating a connection. Unfortunetally, uploading those photos might take a while since the uploading speed is inconsistent. Thus, taking a photo and immediately upload it using the same thread is problematic since we might miss “right angle” photos.

Ergo, the progam manages an uploading queue for pending photos. The SmartFridge class which runs on the main thread, initializes the FileUploader class which starts another thread. After taking a clear photo, the SmartFridge object tell to the FileUploader to upload a file. It pushes the file path to a queue. Then, the other thread gets an item from the queue and uploads it.

file_path = self.__camera.capture()
	if file_path:
		if photoshop.is_photo_clear(file_path):
			photoshop.rotate_photo(file_path)
			self.__fileUploader.upload_file(file_path)
		else:
			os.remove(file_path)

Limited storage volume

The (free) Google Drive storage volume is finate. Hence, we need to remove old files sometimes.

Before uploading new files, we remove the old one if there are too many.

Improvements

There are more features I would like to implement for this project such as:

1. Supporting more than one camera
2. Adding LEDs to the fridge for better lighting
3. Identifing the “right angle” for shooting using software only
4. Dropping blurry photos
5. interpretering the fridge content by machine learning

Fork it

If you want to make your fridge smarter, you can fork the repository easily. Code imporvements are welcome as always.

This is the end result

If you have any questions feel free to comment or contact me directly.

Thank you for reading this. Please share and visit soon again,

Orian Zinger.

Introduction

After graduating from Ben Gurion University (summer 2016) I started working at LiveU which is a high-quality low-delay video streaming units manufacturer.

For example, when you watch the news and see an interview between the news reporter and the news anchor in the studio, the camera that captures the video outdoors, is connected to our unit. When the unit receives the video and audio, it compresses the data as much as it can without compromising the quality. Then, by using all available network links (8 cellular modems, 2 ethernet sockets, and Wi-Fi) and a smart bonding algorithm, we send the data over the LiveU Reliable Transport (LRT) protocol to the server. When the data arrives at the server, it decodes the video and does lots of video-related features such as scaling, closed-captions, ancillary data, stream to CDN, image overlay, and more, and eventually, when the time is right (according to the user’s delay-selection), it sends the video to the monitor.

LRT enables reliable, low delay live video streaming over non-guaranteed IP networks and ensures reliable transport over the most unreliable of networks, including cellular networks – allowing you to acquire exciting live content in even the most remote locations.

We develop the hardware and the software for our products mainly in Python, C++, and Java.

Features

I arrived at the Application and Connectivity team in the R&D department and started learning about the complex system architecture and Python. Soon, I started fixing bugs and learned how to use the test framework. I’ve developed many features. I’ll elaborate on the top five:

Metadata

I created NewsML-G2 based Metadata for the videos and files. Since I developed the feature end to end and the Metadata is collected from lots of places in the software, it made me understand how different processes/threads/classes work together and their responsibility.

LiveU MOS/ENPS Connector

By creating a proxy server that connects our backend server and the ENPS server, I’ve learned a lot about the application environment and infrastructure. For example, configuring syslog, creating a watchdog for the application, and more.

IFB

One of the interesting features I’ve developed is the new generation of Interruptible Feedback (IFB) communication between the unit and the server. This feature was developed by Dima Danilov and me. We had a wonderful time together. I learned a lot from him and from implementing a VoIP application using SIP protocol.

Point to Multipoint

We use it when we want a server to distribute the data to other servers using the LRT protocol. The feature was written for POC and had lots of issues. Thus, before adding new capabilities, I started debugging the code and fixing it. Then, I modified the product-engine so it will send much fewer data and I also reduced the delay from 0.8 seconds to 0.15 seconds. By doing this, I achieved cost reduction for the P2MP operation.

Ultra-high-speed file transfer

There are clients that sometimes shoot events (for example, sports events) with more than one camera so they can have multiple angles of the same event. They don’t have lots of LiveU units, so they want to upload the footage after filming them. Since the video quality is superb and the footage is pretty long, the files are usually huge (dozens of gigs). They requested us to implement such a feature and fortunately, I got the task. I designed the procedure end-to-end and developed it along with four more engineers.

Since the units are running on an ARM chipset, we utilize the limited resources (CPU, memory, disk space, etc.) as much as we can for reaching the best upload speed we can achieve which is more than 200 Mbps.

Promotion

After approximately two and a half years at LiveU, I was promoted to be a technical leader. As a TL, I didn’t manage other colleagues directly, but I’ve mentored new (juniors and seniors) developers and lead software-projects by doing design reviews and code reviews.

Conclusion

LiveU is a great company with awesome people, an interesting line of products, and cool technologies, but after working there four years I decided that the best for my career is facing other challenges and learning more technologies.

The next step for my career includes revolutionizing cloud services. I’m super excited about this role and can’t wait to tell you guys about the new quirky bugs I’ll encounter. For more information about the new role, check the About page.

If you have any questions feel free to comment or contact me directly.

Thank you for reading this. Please share and visit soon again,

Orian Zinger.

It’s the combination of boring practical things with simple, but modern/futuristic improvement that makes me love this realm. I admit. I’m a geek. I like those stuff. When a new microcomputer is available, I compare it to other microcomputers, and sometimes I even purchase it. I use them in my IoT projects.

What is IoT

Almost all our electrical appliances at home have a sort of microcomputer. Some can control heat. Some can clean our clothes. Each appliance is stand-alone. It’s there. It can do X, Y, and Z but nothing more. It (in most cases) doesn’t support software upgrades too.

Once we connect the appliance to the internet, we get a new interface to this device. Suddenly we can control it without pressing its buttons. We can get some information regarding the operations, and also automate the machine to do something for us. For example, I can tell my robotic vacuum cleaner to start working every day after I leave the house. Finally, we can upgrade the machine to an advanced version and get new features and fewer bugs (I hope).

IoT is an acronym for “Internet of Things”. It’s the name for those devices (“things”) that are now connected to the internet.

For example, connecting the AC to the internet can help us by turning it on before we get home so our place will be warm and cozy (or cold and refreshing) before we arrive.

Refrigerator

I assume you already know what a refrigerator does, so let’s talk about how smart it is.

Modern devices can do lots of smart things including connecting to the local WiFi network, but usually, they’re very expensive.

What if I already have an excellent fridge? It’s efficient, reliable, practical and looks nice. I don’t want to replace it with an expensive one, right? I just want to make it smarter. I want to connect it to the internet. This is what I did. Sort of.

You see, I can’t reach the motherboard of the fridge and a WiFi dongle. Thus, I bought a Raspberry Pi (RPi) microcomputer. It’s a tiny device that runs Linux OS (and Windows 10 for IoT) and has few IO interfaces (like USB sockets, WiFi interface, and more).

Since it runs Linux, I can install Python and run code on this small device.

Great. I can run Python code on a computer. What’s the big deal? Where is the Smart fridge thingy?

Motivation

When I go to the supermarket to buy some groceries, I (sometimes) forget what’s going on in my fridge. It makes me think very hard. Do I have lettuce? Do I need more cheese? Did I finish the onions? What about eggs?

Usually, I prefer to buy and find out that I already have tons of onions instead of not buying them at all. This procedure overloads my fridge and put pressure on my tummy to eat the food since I don’t like throwing food away. If I could only know what’s going on in my fridge when I buy those things. How can I take a glimpse on my fridge while I’m not at home (and no one else is there)?

Smart Fridge

By attaching a small camera to the fridge’s door, and connecting it to the RPi, I achieved a digital eye that captures pictures of my fridge content every time I open it. Then, I connected my RPi to the local Wi-Fi network and wrote some code for uploading the pictures to my Google Drive.

Since the application is also installed on my smartphone, I can browse those photos while I’m away.

Note that I bought an ultra-wide camera lens, so it captures the entire fridge in one shot.

In the next article, I’ll elaborate on the software design, issues I encountered, and more. Stay tuned!

If you have any questions feel free to comment or contact me directly.

Thank you for reading this. Please share and visit soon again,

Orian Zinger.

Some practices are considered better than others.

 best practice is a method or technique that has been generally accepted as superior to any alternatives because it produces results that are superior to those achieved by other means or because it has become a standard way of doing things

Wikipedia

Bad practice is an unwise thing to do

Merriam-Webster

Anti-patterns are design patterns that are considered a bad practice and we (in most cases) should avoid using it.

One of the popular infamous anti-patterns is the singleton pattern.

This pattern restricts the user from creating more than one instance of a class. Sometimes we need only one instance of a class, then why is it an unwise thing to do?

First, restrictions limit our software design and implementations. In some cases, the class can be initialized more than once, but the pattern makes us implement the code differently.

Second, using singleton is like declaring a non-const global variable because anyone can access it. It makes debugging and maintaining the code hard since it reduces the modularity and flexibility of the program. We should try to minimize the variable scope.

Third, in multi-threaded applications, we have to verify that the instance is thread-safe (no deadlocks, no starvations, and obviously, no race conditions). Using mutex/semaphore/atomic/lock will slow the program.

As I mentioned before, rarely an anti-pattern is exactly what we need, but I suggest thinking twice before choosing such a pattern.

References

1. https://www.tutorialspoint.com/why-are-global-variables-bad-in-c-cplusplus

If you have any questions feel free to comment or contact me directly.

Thank you for reading this. Please share and visit soon again,

Orian Zinger.

Creation

First, we need to define a new AST node in the ASDL file:

cmpop = Eq | NotEq | Lt | LtE | Gt | GtE | Is | IsNot | In | NotIn | Isnt

Second, we need to regenerate the AST code using make regen-ast

Usage

We finished creating the new AST. Now, let’s use it in the grammar file:

isnt_bitwise_or[CmpopExprPair*]: 'isnt' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, Isnt, a) }

Then, we need to regenerate the parser code using make regen-pegen

Compilation

You probably know that Python is a dynamically typed language which uses an interpreter for running the code instead of a compiler, like lots of other programming languages, and yet, the title is “compilation”.

What is Python bytecode?

Reading the Python code, creating tokens, and parsing them takes time. Thus, the interpreter tries to optimize this process by creating a lower-level language upon module importing.

When we ask the interpreter to import a module named mymodule, it searches the mymodule.pyc file.

If no such file exists, or exists but the writing timestamp of the mymodule.py file is advanced compared to the compiled file, it runs the code regularly and saves the bytecode in mymodule.pyc. Otherwise, it uses the compiled file instead of reading the code and this saves time.

So far, we created a new AST and told the parser to use it, but now, we should explain the compiler what should it do when encountering such an AST:

    case Is:
        ADDOP_I(c, IS_OP, 0);
        return 1;
    case IsNot:
    case Isnt:
        ADDOP_I(c, IS_OP, 1);
        return 1;

As you can see, the Isnt AST shares the same bytecode as the IsNot. They both use the IS_OP operation, but the difference between the Is AST and these ASTs, is the last number. Eventually, the evaluator that runs the bytecode has no clue what is not or isnt keywords are. It compares the addresses of the operands and “xors” the result with the provided input (0 or 1).

The evaluation code is located in Python/ceval.c:

        case TARGET(IS_OP): {
            PyObject *right = POP();  // right operand
            PyObject *left = TOP();  // left operand
            int res = (left == right)^oparg;  // address comparison and xor with input
            PyObject *b = res ? Py_True : Py_False;  // converting the answer to True/False
            Py_INCREF(b);
            SET_TOP(b);
            Py_DECREF(left);
            Py_DECREF(right);
            PREDICT(POP_JUMP_IF_FALSE);
            PREDICT(POP_JUMP_IF_TRUE);
            FAST_DISPATCH();
        }

Results

The Python’s disassembler has a module called dis. Using this module, we can print bytecodes.

For example, let’s check this simple function

import dis

def foo():
        x = 0
        y = 1
        print(x is y)
        print(x is not y)
        print(x isnt y)


dis.dis(foo)

Output:

# I added titles, so it would be clearer. Take a look after this block
# 1           2     3                    4  5
  4           0 LOAD_CONST               1 (0)
              2 STORE_FAST               0 (x)

  5           4 LOAD_CONST               2 (1)
              6 STORE_FAST               1 (y)

  6           8 LOAD_GLOBAL              0 (print)
             10 LOAD_FAST                0 (x)
             12 LOAD_FAST                1 (y)
             14 IS_OP                    0
             16 CALL_FUNCTION            1
             18 POP_TOP

  7          20 LOAD_GLOBAL              0 (print)
             22 LOAD_FAST                0 (x)
             24 LOAD_FAST                1 (y)
             26 IS_OP                    1
             28 CALL_FUNCTION            1
             30 POP_TOP

  8          32 LOAD_GLOBAL              0 (print)
             34 LOAD_FAST                0 (x)
             36 LOAD_FAST                1 (y)
             38 IS_OP                    1
             40 CALL_FUNCTION            1
             42 POP_TOP
             44 LOAD_CONST               0 (None)
             46 RETURN_VALUE

Titles:

1. Line number of original code
2. Bytecode address offset
3. Opcode
4. Opcode’s argument
5. Human-friendly explanation

In bytecode address 32, we start the last line of the function. In bytecode address 38, we use the IS_OP opcode with 1 as its argument. similar to the is not keyword

You can find in my commit these changes and some more (i.e. test, documentation).

References

1. https://stackoverflow.com/questions/15839555/when-are-pyc-files-refreshed
2. https://stackoverflow.com/questions/3299648/python-compilation-interpretation-process

If you have any questions feel free to comment or contact me directly.

Thank you for reading this. Please share and visit soon again,

Orian Zinger.

Let’s review this code:

#include <iostream>
#include <string>


class DataStructure {
public:

	DataStructure (uint64_t size) : m_size(size), m_data(new uint64_t[m_size])
	{
		print("ctor");
	}

	~DataStructure ()
	{
		print("dtor");
		if (m_data)
		{
			delete[] m_data;
			m_data = nullptr;
		}
	}

	DataStructure (const DataStructure & other) : m_size(other.m_size), m_data(new uint64_t[m_size])
	{
		print("copy ctor");
		std::copy(other.m_data, other.m_data + m_size, m_data);
	}

	DataStructure & operator = (const DataStructure & other)
	{
		print("operator =");
		if (this == &other)
			return *this;
		delete [] m_data;
		m_data = nullptr;
		DataStructure tmp(other);
		std::swap(m_size, tmp.m_size);
		std::swap(m_data, tmp.m_data);
		return *this;
	}

private:
	uint64_t	m_size;
	uint64_t *	m_data;

	void print(const std::string & message)
	{
		std::cout << this << " " << message << std::endl;
	}
};

I’ve implemented a data structure that holds two members: m_data and m_size. Since we have to free the allocated memory, I implemented a destructor. This class also contains a copy constructor and a copy assignment operator as recommended by the rule of three.

Let’s run this code:

int main()
{
	DataStructure ds(10);
	return 0;
}

Output:

0x7ffe19590f00 ctor
0x7ffe19590f00 dtor

It’s pretty obvious. Let’s assign to our ds variable a new DataStructure instance:

int main()
{
	DataStructure ds(10);
	ds = DataStructure(20);
	return 0;
}

Output:

0x7fffcd714f30 ctor  // make sense (same as previous output)
0x7fffcd714f40 ctor  // make sense (we initialize another object)
0x7fffcd714f30 operator =
0x7fffcd714ed0 copy ctor  // mmm... why? we already initialized an object
0x7fffcd714ed0 dtor
0x7fffcd714f40 dtor
0x7fffcd714f30 dtor

On line #2 we initialize an object (rvalue), but we initialize another temporary object on line #4. In this code, we initialize two DataStructure objects, but actually, we called the constructors 3 times. Memory allocation wise, We pay more than we get.

The move semantics came to fix this resource waste by implementing move constructor and move assignment operator:

DataStructure (const DataStructure && other)
{
	print("move copy ctor");
	m_size = other.m_size;
	m_data = other.m_data;
	other.m_size = 0;
	other.m_data = nullptr;
}

DataStructure & operator = (const DataStructure && other)
{
	print("move operator =");
	if (this == &other)
		return *this;
	m_size = other.m_size;
	m_data = other.m_data;
	other.m_size = 0;
	other.m_data = nullptr;
	return *this;
}

Output:

0x7ffcb5bc9660 ctor
0x7ffcb5bc9670 ctor
0x7ffcb5bc9660 move operator =
0x7ffcb5bc9670 dtor
0x7ffcb5bc9660 dtor

Once we assign an expression to the variable, the compiler checks the expression’s value category. If it’s an rvalue reference, it calls the move assignment operator instead of the regular assignment operator. By doing so, we can swap the members. We don’t care about the rvalue reference members since we know it’s a temporary variable that soon will be dead, but it’s still important to keep our program in a valid state. Thus, I reset the members to 0 and nullptr.

Other move-semantics-related standard functions

Under the <utility> header, we can find two new functions:

std::move

This function receives one parameter. What does it move? Nothing. The name confused me at first, but it doesn’t move anything anywhere.

The function receives an lvalue expression, creates an xvalue expression, and returns an rvalue reference.

It’s good when you want to let the compiler know that a variable will die soon, so it can be moved. The compiler will try to move it instead of copy it to another variable when needed, but it doesn’t guarantee the movement. Maybe renaming the function to std::enable_movement would be better for understanding this function.

For example:

int main()
{
	DataStructure ds(10);
	DataStructure ds2(std::move(ds));
	return 0;
}

Output:

0x7ffc78fe7dc0 ctor
0x7ffc78fe7dd0 move copy ctor
0x7ffc78fe7dd0 dtor
0x7ffc78fe7dc0 dtor

std::forward

This function also receives one parameter. Does it forward it? Yes, this function forwards the parameter to another function keeping the original value category. It’s useful when implementing a template function that receives a T&& parameter. The parameter looks like an rvalue reference expression, but it can receive also lvalue expressions. Thus, when we want to forward this parameter to another function, we would like to tell the compiler to keep the original value category.

For example, check this generic factory function that receives a parameter and duplicates it:

template <class T>
std::shared_ptr<T> duplicate(T&& parameter)
{
	return std::make_shared<T>(parameter);
}

int main()
{
	DataStructure ds(10);
	auto ds2 = duplicate(std::move(ds));
	return 0;
}

Output:

0x7fff455b6ea0 ctor
0x55f300cd8330 copy ctor
0x55f300cd8330 dtor
0x7fff455b6ea0 dtor

In this code, we used std::move, but the compiler didn’t use the move copy constructor.

Now, let’s use the std::forward function:

template <class T>
std::shared_ptr<T> duplicate(T&& parameter)
{
	return std::make_shared<T>(std::forward<T>(parameter));
}

Output:

0x7ffd5d1a2070 ctor
0x55f16210e330 move copy ctor
0x55f16210e330 dtor
0x7ffd5d1a2070 dtor

References:

1. https://en.cppreference.com/w/cpp/utility/move
2. https://stackoverflow.com/questions/3106110/what-is-move-semantics
3. https://www.internalpointers.com/post/c-rvalue-references-and-move-semantics-beginners
4. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2006/n2027.html#Perfect_Forwarding

If you have any questions feel free to comment or contact me directly.

Thank you for reading this. Please share and visit soon again,

Orian Zinger.

In C++03 (and before), we had two value categories:

1. lvalue
2. rvalue

In C++11 (and later), we were introduced with new categories:

1. lvalue
2. rvalue – prvalue or xvalue
3. xvalue
4. glvalue – lvalue or xvalue
5. prvalue – C++03 rvalue

In this article, I’ll explain the categories and in the next article, I’ll elaborate on why we need those new categories (spoiler alert: std::move). Stay tuned!

lvalue VS (rvalue (C++03) or prvalue(C++11))

The simple way of understanding the difference is keeping in mind that during the code execution, an lvalue is stored in the memory/register while an rvalue is temporary.

Let’s review this code:

int n = 0;

In this case, we’re declaring an integer and assigning a value. This variable is stored in the stack section of the memory. This is the lvalue. 0 has no address, it’s just a literal constant. This is the rvalue.

In x86 architecture it looks like this:

push	rbp
mov		rbp, rsp
mov		DWORD PTR -4[rbp], 0
pop		rbp

The Register Stack Pointer (rsp) is the pointer to the top of the stack section in the memory.

In line #3 we write 0 to the rsp address.

The assignment rule

The assignment requires an lvalue on the left and an rvalue on the right. It means that swapping the lvalue and rvalue (0 = n;) causes a compilation error.

Let’s review this code:

int x = 0;
int y = x;

In line #2 we use x as an rvalue for the assignment, but x is an lvalue. In cases like this, a conversion of lvalue -> rvalue is performed.

Functions

Let’s review this code:

int number = 0;

int get_number()
{
	return number;
}

int main()
{
	get_number() = 1;
	return 0;
}

Take a minute and think what the compiler will complain about.

…

The get_number() function returns an integer. It means that it copies the return value (number) and returns an rvalue. In line #10, we assign the rvalue 1 to another rvalue in contrary to the assignment values rule.

How can we fix it?

Value referencing

Instead of returning the integer, we can return a reference to the integer. Reference is an alias to another datum so the program can access it indirectly. Thus, a reference to an lvalue is an lvalue as well.

In this case, we assign the rvalue 1 to the lvalue get_reference_to_number():

int number = 0;

int & get_reference_to_number()
{
	return number;
}

int main()
{
	get_reference_to_number() = 1;
	return 0;
}

Note: I don’t recommend setting new values using this way unless it’s a special case or it’s used for setting variables in a customized data structure like data_structure[index] = new_value;

If we want to set a value, we probably want to create a setter function instead of a reference getter function like above since it’s easier to read and to understand.

Referencing a parameter

Let’s review this code:

void foo(int & number)
{}

int main()
{
	foo(1);
	return 0;
}

The argument we send is 1, an rvalue. The function expects to receive a reference to an integer, an lvalue. The compiler will, of course, complain about it. We can fix it by declaring a variable (an lvalue) right before calling the function, but we don’t want to create a local variable each time just for that, do we?

Const referencing

By declaring the parameter to be const, the compiler allocates space for this value. Then, when we store the rvalue, we have an lvalue which we can transfer by reference to the function.

void foo(const int & number)
{}

int main()
{
	foo(1);
	return 0;
}

It works, of course, but we can’t change the value of the parameter inside foo. What if we do want to change the value?

rvalue reference

I mentioned earlier the prvalue is temporary data. How can we refer to a temporary variable?

One of the big changes in C++11 is the rvalue reference. Now, we can tell the compiler we need the specified rvalue a little longer. Most of the use cases are related to the move semantics (next article, remember?), but meanwhile, I can suggest this example:

void foo(int && number)
{
	number++;
}

int main()
{
	foo(1);
	return 0;
}

Notice the && notation in the function signature. By declaring number to be an rvalue reference, the compiler allocates memory for the rvalue 1 and now we can change it.

The standard doesn’t specify the memory section for saving rvalue references.

xvalue

We understand the difference between rvalue, lvalue, glvalue, and prvalue, but what is the xvalue all about?

The expression’s value category is an xvalue (eXpiring value) if it is near the end of its lifetime and it’s movable. For example, an rvalue reference.

References:

1. https://www.internalpointers.com/post/understanding-meaning-lvalues-and-rvalues-c
2. https://eli.thegreenplace.net/2011/12/15/understanding-lvalues-and-rvalues-in-c-and-c

If you have any questions feel free to comment or contact me directly.

Thank you for reading this. Please share and visit soon again,

Orian Zinger.