In a previous post, I looked at ways of measuring frequency accurately using an RP2040 CPU programmed in C. In this project, I have re-coded those functions in MicroPython, and provided a few enhancements.

I use Direct Memory Access (DMA) to minimise the CPU workload, and avoid the need for interrupts, but this raises a problem: when programming in C, there is a substantial Application Programming Interface (API) that allows everything on the RP2040/2350 chip to be configured using simple function calls. However, MicroPython aims to be more-or-less compatible with a wide range of CPUs, so not much CPU-specific code has been included. This means that we have to do some low-level programming if we want to use the more obscure functions of the Pulse Width Modulation (PWM) peripheral.

My first attempt at ADC and DMA low-level programming used MicroPython ‘uctypes’ to create a model of the peripherals, and if you are interested in the nuts-and-bolts of this method, I suggest you browse the post here.

This approach produced excellent results, but wasn’t very ‘pythonic’, so I’ve encapsulated common API functions in Python classes; when choosing function names, I’ve copied those in the C API when possible. So for example, setting the PWM peripheral in C is:

#define PWM_GPIO_PIN 3
uint slice = pwm_gpio_to_slice_num(PWM_GPIO_PIN);
pwm_config cfg = pwm_get_default_config();
pwm_config_set_clkdiv_mode(&cfg, PWM_DIV_B_RISING);
// ..and any other settings, then..
pwm_init(slice, &cfg, false);

In MicroPython this is done by instantiating a device from the PWM class:

import pico_devices as devs
PWM_GPIO_PIN = 3
pwm = devs.PWM(PWM_GPIO_PIN)
pwm.set_clkdiv_mode(devs.PWM_DIV_B_RISING)
# ..and any other settings..

Not only are the function call names very similar between C and Python, but also the underlying code is similar; for example, the set_clkdiv function is defined as:

class PWM:
    def set_clkdiv_mode(self, mode):
        self.slice.CSR.DIVMODE = mode

where ‘slice’ is the (somewhat unusual) name for a PWM channel, CSR is the Control and Status Register, and DIVMODE is a 2-bit field within that register. If you want to learn more about the internal structure of the PWM peripheral, I suggest you take a look at the C language post.

The Python classes have been kept very ‘lean’, with no error-checking of the parameters; when time permits, I intend to create a sub-class that provides comprehensive parameter-checking .

The initial version of the post was RP2040-only; for the RP2350 CPU on the Pico2 the underlying logic is the same, but a few details had to be changed, as detailed towards the end of this post.

Test signal

We need a waveform for the tests, which the RP2040 PWM peripherals can easily provide. The following code generates a 100 kHz square wave on GPIO pin 4:

PWM_OUT_PIN = 4             # GPIO pin for output
PWM_DIV = 125               # 125e6 / 125 = 1 MHz
PWM_WRAP = 9                # 1 MHz / (9 + 1) = 100 kHz
PWM_LEVEL = (PWM_WRAP+1)//2 # 50% PWM

# Start a PWM output
def pwm_out(pin, div, level, wrap): 
    devs.gpio_set_function(pin, devs.GPIO_FUNC_PWM)
    pwm = devs.PWM(pin)
    pwm.set_clkdiv_int_frac(div, 0)
    pwm.set_wrap(wrap)
    pwm.set_chan_level(pwm.gpio_to_channel(pin), level)
    pwm.set_enabled(1)
    return pwm

test_signal = pwm_out(PWM_OUT_PIN, PWM_DIV, PWM_LEVEL, PWM_WRAP)

The choice of GPIO pin number is quite arbitrary, you just need to be aware when programming PWM that there are are 16 channels (technically 8 slices, each with 2 channels) mapped onto 32 pins, so if I’m using GPIO pin 4 for this signal, I can’t use PWM on GPIO 20. However, that pin is still available for any other purpose, so the limitation usually isn’t a problem.

16-bit counter

The RP2040 PWM peripheral can also act as a pulse counter, and by counting the number of edges in a given time, we can establish the frequency. This pulse input capability is only available on odd GPIO pin numbers (i.e. channel B of a PWM ‘slice’).

The code is quite simple; it is only necessary to set the mode, and the frequency divisor:

# Initialise PWM as a pulse counter (gpio must be odd number)
def pulse_counter_init(pin, rising=True):
    devs.gpio_set_function(pin, devs.GPIO_FUNC_PWM)
    ctr = devs.PWM(pin)
    ctr.set_clkdiv_mode(devs.PWM_DIV_B_RISING if rising else devs.PWM_DIV_B_FALLING)
    ctr.set_clkdiv(1)
    return ctr

Now we can make a simple frequency meter, by clearing the count to zero, then enabling the counter for a specific period of time:

# Enable or disable pulse counter
def pulse_counter_enable(ctr, en):
    if en:
        ctr.set_ctr(0)
    ctr.set_enabled(en)

# Get value of pulse counter
def pulse_counter_value(ctr):
    return ctr.get_counter()

pulse_counter_enable(counter, True)
time.sleep(0.1)
pulse_counter_enable(counter, False)
val = pulse_counter_value(counter)
print("Sleep 0.1s, count %u" % val)

The count-value for a 100 kHz signal and a 100 millisecond sleep is theoretically 10000, but is actually around 10015, due to the time-delays associated with the Python instructions. I’ll be describing ways to eliminate these delays later on in this post.

32-bit counter

Unfortunately the PWM peripheral has only a 16-bit counter, which can be too short for high-frequency signals or long sample-times. In the C code I polled the count value, to count the number of times it rolled over past 65535, then add on that number to the final result.

An alternative method is to take advantage of the fact that the DMA controller has a 32-bit down-counter that decrements every time a DMA cycle is completed. So we just need to set the DMA count to a large number, and set the PWM peripheral to trigger a DMA cycle on every rising or falling edge of the input signal.

This prompts the question “what data should the DMA transfer?” and the answer is “anything: it doesn’t matter”, but we still have to be careful to ensure the transfers don’t over-write some random area in memory. We need to very carefully specify the DMA source & destination addresses, using the binary ‘array’ data type, which occupies a fixed area of memory, without all the complexities of a Python ‘object’. The syntax to create the array is somewhat unusual; rather than just specifying a size, we have to provide the initial data using an iterator. The pico_devices library provides a helper function to simplify the process:

# Create 32-bit array (to receive DMA data)
def array32(size):
    return array.array('I', (0 for _ in range(size)))

To use this array with DMA, we have to get its address in memory, using ‘uctypes.addressof’:

# Get address of variable (for DMA)
def addressof(var):
    return uctypes.addressof(var)

Armed with these functions, we can initialise the DMA controller:

ext_data = devs.array32(1)  # Dummy array for extended counter

# Use DMA to extend pulse counter to 32 bits
def pulse_counter_ext_init(ctr):
    ctr.set_enabled(False)
    ctr.set_wrap(0)
    ctr.set_ctr(0)
    ctr_dma = devs.DMA()
    ctr_dma.set_transfer_data_size(devs.DMA_SIZE_8)
    ctr_dma.set_read_increment(False)
    ctr_dma.set_write_increment(False)
    ctr_dma.set_read_addr(devs.addressof(ext_data))
    ctr_dma.set_write_addr(devs.addressof(ext_data))
    ctr_dma.set_dreq(ctr.get_dreq())
    ctr.set_enabled(True)
    return ctr_dma

We’re modifying the PWM 16-bit counter to wrap around to zero on every input edge, and initialising its starting value to zero, which is essential. Then a DMA channel is instantiated, and configured to copy 8 bits from the data array back to the data array.

It is important to note that enabling the DMA counter does not necessarily start the transfer from scratch; if a transfer has already started, the controller will resume counting where it left off. So it is necessary to clear out any existing transfer, before stating a new one:

# Start the extended pulse counter
def pulse_counter_ext_start(ctr_dma):
    ctr_dma.abort()
    ctr_dma.set_trans_count(0xffffffff, True)
 
# Stop the extended pulse counter
def pulse_counter_ext_stop(ctr_dma):
    ctr_dma.set_enable(False)
 
# Return value from extended pulse counter
def pulse_counter_ext_value(ctr_dma):
    return 0xffffffff - ctr_dma.get_trans_count()

Using the extended pulse counter is similar to the 16-bit counter, we just need to remember the limitation that if the 32-bit value is exceeded, the counter will stop, and not wrap around.

counter_dma = pulse_counter_ext_init(counter)
pulse_counter_ext_start(counter_dma)
time.sleep(1.0)
val = pulse_counter_ext_value(counter_dma)
pulse_counter_ext_stop(counter_dma)
print("Sleep 1.0s, ext count %u" % val)

A typical response is:

Sleep 1.0s, ext count 100011

This shows that the count is not limited to 16 bits.

Gating a counter

To accurately count pulses with a specific time-frame, it is necessary for the timing to be accurate, and as we have seen, the ‘sleep’ function introduces a significant error. To eliminate this, we need a hardware mechanism whereby a timer directly enables and disables the counter (a process called ‘gating’) without requiring any intervention from the CPU.

This involves programming another PWM channel to act as a timer, then when the time has expired, using DMA to modify the counter’s register to stop counting. The default PWM clock is 125 MHz, the prescaler is 8 bits, and the counter register is 16 bits, effectively 17 bits if we engage ‘phase correct’ mode. So the slowest gate frequency is 125e6 / (256 * 65536 * 2) = 3.725 Hz, a gate-time of 0.268 seconds; I’ve opted for 0.25 seconds.

GATE_TIMER_PIN = 0          # Used to define PWM slice
GATE_PRESCALE = 250         # 125e6 / 250 = 500 kHz
GATE_WRAP = 125000          # 500 kHz / 125000 = 4 Hz (250 ms)

# Initialise PWM as a gate timer
def gate_timer_init(pin):
    pwm = devs.PWM(pin)
    pwm.set_clkdiv_int_frac(GATE_PRESCALE, 0)
    pwm.set_wrap(int(GATE_WRAP/2 - 1))
    pwm.set_chan_level(pwm.gpio_to_channel(pin), int(GATE_WRAP/4))
    pwm.set_phase_correct(True)
    return pwm

This code uses GPIO pin 0 to identify which PWM ‘slice’ is being used. Since we aren’t enabling PWM I/O on that pin, it can still be used for any other function, such as serial output.

We need to trigger a DMA cycle when the gate PWM times out; that cycle will be used to disable the counter PWM. So when initialising the DMA channel, we need to capture the non-enabled state, that will be written into the counter CSR register; this is only one 32-bit value, but I’m using a fixed array for that value, since DMA can’t handle the complexities of Python object storage.

gate_data = devs.array32(1)

# Initialise gate timer DMA
def gate_dma_init(ctr, gate):
    dma = devs.DMA()
    dma.set_transfer_data_size(devs.DMA_SIZE_32)
    dma.set_read_increment(False)
    dma.set_write_increment(False)
    dma.set_dreq(gate.get_dreq())
    gate_data[0] = ctr.slice.CSR_REG
    dma.set_read_addr(devs.addressof(gate_data))
    dma.set_write_addr(ctr.get_csr_address())
    return dma

To start the frequency measurement, it is necessary to set the DMA count (since this is reset by a DMA cycle), enable DMA, then enable the gate & counter PWM devices simultaneously.

# Start frequency measurment using gate
def freq_gate_start(ctr, gate, dma):
    ctr.set_ctr(0)
    gate.set_ctr(0)
    dma.set_trans_count(1, True)
    ctr.set_enables((1<<ctr.slice_num) | (1<<gate.slice_num), True)

If all is well, the test signal frequency should be reported correctly on the console:

Gate 250.0 ms, count 25000, freq 100.0 kHz

Edge timer

An alternative method for measuring frequency is to measure the times between the rising or falling edges, producing values that are the reciprocal of the frequency. This is particularly useful when dealing with slow signals, or if you want to implement a method to eliminate ‘rogue’ pulses, since you get one measurement for each cycle of the input signal, so could reject pulses that are outside the expected frequency range.

By now, you won’t be surprised to learn that I use DMA to capture the time-value for each edge; there is a convenient 32-bit microsecond-value that can be copied into a suitably-sized array on edge positive or negative edge; the sole function of the PWM peripheral is to detect the edge, and trigger a DMA cycle.

We’ll be generating a test signal as before, but this time it is 10 Hz:

PWM_DIV = 250               # 125e6 / 125 = 500 kHz
PWM_WRAP = 50000 - 1        # 500 kHz / 50000 = 10 Hz

The PWM peripheral initialisation is similar to previous functions:

# Initialise PWM as a timer (gpio must be odd number)
def timer_init(pin, rising=True):
    devs.gpio_set_function(pin, devs.GPIO_FUNC_PWM)
    pwm = devs.PWM(pin)
    pwm.set_clkdiv_mode(devs.PWM_DIV_B_RISING if rising else devs.PWM_DIV_B_FALLING)
    pwm.set_clkdiv(1)
    pwm.set_wrap(0);
    return pwm

The DMA controller is also similar, but the destination is set to auto-increment with every transfer:

# Initialise timer DMA
def timer_dma_init(timer):
    dma = devs.DMA()
    dma.set_transfer_data_size(devs.DMA_SIZE_32)
    dma.set_read_increment(False)
    dma.set_write_increment(True)
    dma.set_dreq(timer.get_dreq())
    dma.set_read_addr(devs.TIMER_RAWL_ADDR)
    return dma

Starting the DMA is a bit different; firstly we need to use the ‘abort’ command to stop any previous transfer that might still be in progress. If we didn’t do that, and just enabled DMA, the transfer would resume where it left off – the new settings would be ignored. Secondly we need to set both the transfer count and the destination address, since these will have been changed by a previous transfer.

# Start frequency measurment using gate
def timer_start(timer, dma):
    timer.set_ctr(0)
    timer.set_enabled(True)
    dma.abort()
    dma.set_write_addr(devs.addressof(time_data))
    dma.set_trans_count(NTIMES, True)

The main program needs to perform some simple maths to derive the frequency, guarding against the possibility that there may be insufficient time-values (1 or less):

NTIMES = 9                       # Number of time samples
time_data = devs.array32(NTIMES) # Time data

timer_pwm = timer_init(PWM_IN_PIN)
timer_dma = timer_dma_init(timer_pwm)
    
timer_start(timer_pwm, timer_dma)
time.sleep(1.0)
timer_stop(timer_pwm)
    
count = NTIMES - timer_dma.get_trans_count()
data = time_data[0:count]
diffs = [data[n]-data[n-1] for n in range(1, len(data))]
total = sum(diffs)
freq = (1e6 * len(diffs) / total) if total else 0
print("%u samples, total %u us, freq %3.1f Hz" % (count, total, freq))

The frequency of the test signal should be displayed on the console:

9 samples, total 800000 us, freq 10.0 Hz

Running the code

The source files are available on Github here. It is necessary to load the library file ‘pico_devices.py’ onto the target system; if you are using the Thonny editor, this is done by right-clicking the filename, and selecting ‘Upload to /’. You can then run one of the program files to measure frequency:

• pico_counter.py: use simple pulse counting and sleep timing
• pico_freq.py: use DMA to accurately gate the pulse counter
• pico_timer.py: use time-interval (reciprocal) measurement

Don’t forget to link GPIO pin 4 (the test signal output) to pin 3 (the measurement input) for these tests.

Pico2 RP2350 processor

Some minor changes had to be made to accommodate the newer processor; for simplicity, the code in pico_devices.py auto-detects which platform it is running on, and alters the settings accordingly.

Identifying the CPU

There is a Micropython library function uos.uname() that provides CPU identification. Here is the output, running on the latest Micropython versions at the time of writing (the full Pico2 release isn’t available yet):

# Response on RP2040 Pico
(sysname='rp2', nodename='rp2', release='1.23.0', version='v1.23.0 on 2024-06-02 (GNU 13.2.0 MinSizeRel)', machine='Raspberry Pi Pico with RP2040')

# Response on RP2350 Pico2
(sysname='rp2', nodename='rp2', release='1.24.0-preview', version='v1.24.0-preview.321.g0ff782975 on 2024-09-23 (GNU 14.1.0 MinSizeRel)', machine='Raspberry Pi Pico2 with RP2350')

Since the code should be compatible with any board that uses an RP2350, I decided to set a boolean variable by checking the ‘machine’ string for the number ‘2350’, e.g.

PICO2 = "2350" in uos.uname().machine

Clock frequency

The default clock frequency for the Pico2 is 150 MHz, as opposed to 125 MHz for the RP2040. This increase highlights an unfortunate problem with the divisor values I’ve previously chosen; to get a 250 ms gate-time, I prescaled the clock frequency by 250 to get 500 kHz, then divided by 125000 to get 4 Hz. However, this doesn’t work with the higher clock frequency, since the prescaler is restricted to an 8-bit value, and the divisor to a 17-bit value (since I’m using ‘phase correct’ mode); the lowest frequency I can get on the Pico2 is 4.49 Hz, so I’m using 5 Hz:

# Frequency gate settings for RP2350 and RP2040
# Prescale must be < 256, and wrap < 131072 (phase correct mode)
if devs.PICO2:
    GATE_PRESCALE = 250     # 150e6 / 250 = 600 kHz
    GATE_WRAP = 120000      # 600 kHz / 120000 = 5 Hz (200 ms)
else:
    GATE_PRESCALE = 250     # 125e6 / 250 = 500 kHz
    GATE_WRAP = 125000      # 500 kHz / 125000 = 4 Hz (250 ms)
GATE_FREQ = devs.CLOCK_FREQ / (GATE_PRESCALE * GATE_WRAP)
GATE_TIME_MSEC = 1000 / GATE_FREQ

Peripheral changes

Although the structure of the CPU peripherals remains unchanged, there are various changes to the details. The most obvious of these are the base addresses:

if PICO2:
    CLOCK_FREQ      = 150e6
    GPIO_BASE       = 0x40028000
    PAD_BASE        = 0x40038000
    ADC_BASE        = 0x400a0000
    PWM_BASE        = 0x400a8000
    TIMER_BASE      = 0x400b0000
    DMA_BASE        = 0x50000000
else:
    CLOCK_FREQ      = 125e6
    GPIO_BASE       = 0x40014000
    PAD_BASE        = 0x4001c000
    ADC_BASE        = 0x4004c000
    PWM_BASE        = 0x40050000
    TIMER_BASE      = 0x40054000
    DMA_BASE        = 0x50000000

There are also significant changes to constants, such as the DMA requests:

if PICO2:
  DREQ_SPI0_TX,  DREQ_SPI0_RX,  DREQ_SPI1_TX, DREQ_SPI1_RX   = 24, 25, 26, 27
  DREQ_UART0_TX, DREQ_UART0_RX, DREQ_UART1_TX, DREQ_UART1_RX = 28, 29, 30, 31
  DREQ_PWM_WRAP0,DREQ_PWM_WRAP1,DREQ_PWM_WRAP2,DREQ_PWM_WRAP3= 32, 33, 34, 35
  DREQ_ADC                                                   = 48
else:  
  DREQ_SPI0_TX,  DREQ_SPI0_RX,  DREQ_SPI1_TX, DREQ_SPI1_RX   = 16, 17, 18, 19
  DREQ_UART0_TX, DREQ_UART0_RX, DREQ_UART1_TX, DREQ_UART1_RX = 20, 21, 22, 23
  DREQ_PWM_WRAP0,DREQ_PWM_WRAP1,DREQ_PWM_WRAP2,DREQ_PWM_WRAP3= 24, 25, 26, 27
  DREQ_ADC                                                   = 36

Refer to pico_devices.py to see all the changes, but in practice the differences can be ignored, as the values change automatically, based on identification of the CPU.

Copyright (c) Jeremy P Bentham 2024. Please credit this blog if you use the information or software in it.

The Raspberry Pi Pico CPU (RP2040) has a remarkably complex set of peripherals, and this is reflected in the very large number of control registers (1,116).

To debug a C or Python application, it can be very helpful to know the values in these registers; instead of adding ‘print’ calls, or using a heavyweight debugger, you can use a 3-wire connection between the Pico and a Raspberry Pi, and view the state of any register without modifying or disrupting the software. This magic is achieved using the Single Wire Debug (SWD) interface; it is mainly used for reprogramming the Flash memory of the Pico, but can do a lot more – it acts as a transparent window into the I/O subsystems, that is completely independent of the CPU.

This capability can be accessed using software tools such as OpenOCD, GDB, and Eclipse, but I wanted to create something much simpler, and easier to use. The end-result is PicoReg, which is a pure-Python program that runs on any Raspberry Pi. In addition to the SWD interface, it can access the standard System View Description (SVD) file, to give a description of every register in considerable detail.

The end-result is a simple-to-use software tool that gives a valuable insight into the inner workings of the Pico.

Installation

Hardware

Only 3 wires are needed; a ground connection, SWCLK and SWDIO.

Any I/O pins on the Pi could be used; for ease of identification, I’ve chosen BCM pin numbers 20 and 21. These should be connected to the SWCLK and SWDIO pins on the edge of the Pico; keep the wires as short as possible, ideally a 150 mm (6 inches) or less.

The pins are defined at the top of picoreg_gpio.py, so can easily be changed to any others, but you need to be sure there is no conflict with other device drivers, such as serial, SPI etc.

# Clock and data BCM pin numbers
CLK_PIN         = 20
DAT_PIN         = 21

The Pico will need to be powered as normal, for example using a 5V supply into the USB socket, or if the Pico USB interface is not connected, linking the the 5V pin on the Pi to the VSYS pin on the Pico.

Software

There are 2 Python files, and one database file:

• picoreg_gpio.py. The low-level interface code, with a very simple command-line interface.
• picoreg_qt.py. A PyQt application with full GUI, that uses picoreg_gpio as a low-level interface.
• rp2040.svd. The System View Description file provided by the Raspberry Pi organisation, that describes all the peripheral registers in XML format

The GUI translates register names (such as TIMER.ALARM3) into physical addresses (such as 0x4005401c) using the SVD file; if it is missing, the GUI won’t work. The software performs best on a Pi 3 or 4; it will work on earlier devices, but is a bit slower.

The files can be found on github here; copy them to any convenient directory, then install PyQt5 on the Pi. The current version at the time of writing is 5.11.3:

sudo apt update
sudo apt install python3-pyqt5

That is all you need to do!

Running the GUI

Start the GUI from a console:

python3 picoreg_qt.py

The application will respond “Loading rp2040.vsd”, then after a short delay, will show the GUI. You may also see some warning messages from PyQt, but these are harmless.

The controls are:

• Core number. The Pico is a 2-core device, and this allows you to select core 0 or 1. There is no practical difference when using Picoreg to look at I/O, since the peripheral registers are common to both cores.
• Verbose. This allows you to see the SWD messages that Picoreg is sending, and the responses obtained; useful when there are problems with the link.
• Connect. This starts communication with the Pico, and verifies the identity of the RP2040 CPU. When the link is broken, it is necessary to re-connect before doing any register accesses.
• Single. When connected, this button takes a single reading from the highlighted register.
• Run. When connected, this button starts a continuous cycle of reading from the highlighted register, at 5 readings per second.

Note that this initial display is not Raspberry-Pi-specific; it will run on any PC, even under Windows. This allows you to browse the register database on any convenient machine, though the low-level I/O code is Pi-specific (using the RPI.GPIO module), so the SWD code only runs on a Pi.

The upper display is a tree structure, containing the peripherals, registers, and fields within the registers. By default it is alphabetically sorted, click on the ‘base’ header, to sort by address. The lower display shows general information, and a description of the selected register.

If the SVD database file is damaged or missing, Picoreg will report a syntax error; check that the file is in the same directory as picroreg_qy.py, and restart.

Click on ‘connect’ and if all is well, the following will be displayed.

SWD connection restart
DPIDR 0x0bc12477

The Debug Port Identification Register value shows that Picoreg connection has succeeded. The software makes 3 attempts to do this, and if unsuccessful, the most likely cause is incorrect wiring, or the Pico board not being powered up.

You can navigate around the register display using mouse (single or double-click) or keyboard, as is usual for such tree displays.

If you have a new un-programmed Pico, try navigating to the TIMER.TIMELR register, and hit ‘Run’.

You should see a rapidly-changing display of the current timer value; you can then move the cursor to other registers, whilst the data collection is still running; the register value under the cursor will be updated, while previously-accessed register values are static. There is a flashing indication in the bottom-left corner of the window to show that data collection is running.

Debugging an application

Load the following MicroPython application onto the Pico

# Simple test of O/P and ADC
from machine import Pin, Timer, ADC
led = Pin(25, Pin.OUT)
temperature = ADC(4)
timer = Timer()

def blink(timer):
    led.toggle()
    print(temperature.read_u16())

timer.init(freq=2, mode=Timer.PERIODIC, callback=blink)

The on-board LED will flash at 1 Hz, and the console will report the ADC value on the temperature channel.

You can use PicoReg to display the raw ADC value, in the ADC.RESULT register:

The state of the I/O pins is in the SIO.SPIO_IN register:

You can even see activity on the USB link, as the data pins toggle high and low:

Potential issues

Styling

This proved to be a surprising headache; it has been remarkably difficult to get consistent styling. Some of the screenshots were taken on a Pi 3 running Qt 5.11.3, remote-controlled from a PC, using SSH:

export DISPLAY=:0.0
python3 picoreg_qt.py

The others are run directly from a Pi console, and look quite different (and not as nice, in my opinion).

I’ve already used some very limited styling to customise the tree display:

TREE_STYLE    = ("QTreeView{selection-color:#FF0000;} " +
                 "QTreeView{selection-background-color:#FFEEEE;} ")

self.tree.setStyleSheet(TREE_STYLE)

However, after many hours of experimentation, I still haven’t found a way of getting a consistent appearance – feel free to tackle this problem yourself!

Errors

PicoReg uses pure-python code; on the plus side, you get the convenience of not having to install any device-drivers, but on the minus-side the SWD timing is a bit unpredictable, and can be stretched out when a higher-priority task takes control of the CPU. This is particularly noticeable when resizing or repositioning the display window; the software makes 3 attempts to re-establish connection with the Pico, but may time out and disconnect.

This behaviour is harmless, and it is only necessary to click the ‘Connect’ button to re-establish communications.

Console interface

If there are problems running the graphical interface, the low-level drivers in can be run directly from the command line:

python3 picoreg_gpio.py [options] [address]
    options: -v Verbose mode
             -r Repeated access 
    address: hexadecimal address to be monitored

The default address to be monitored is the GPIO input 0xD0000004. The low-level driver can’t access the SVD database, so the address has to be specified in hexadecimal, e.g. to display the timer value:

python3 picoreg_gpio.py -r 0x4005400c
SWD connection restart
DPIDR 0x0bc12477
0x4005400c: 0xefa1b0d0
0x4005400c: 0xefa26046
0x4005400c: 0xefa30fc8
0x4005400c: 0xefa3bf32
0x4005400c: 0xefa46ec3
..and so on, use ctrl-C to exit

How it works

The SWD link has 2 wires: a clock line, and a bi-directional data line. All transactions are initiated by the Pi, the Pico only transmits on the data line when requested. The data is sent LSB (Least Significant Bit) first.

Each message starts with an 8-bit header, and the Pico responds with a 3-bit acknowledgement; if this is OK (bit value 1, 0, 0) then a 32-bit data value is transferred (sent or received), followed by a parity bit.

As there is only one data line, the Pi has to switch its direction from transmit to receive to get the acknowledgement, and then switch back to transmit (if it is a write cycle) or allow the Pico to send 32 bits of data (if it is a read cycle).

An extra complication that isn’t emphasised in the ARM documentation, is that the active edge changes as well; the Pi sends data that is stable on the rising clock edge, but the Pico data is stable on the falling clock edge, as shown in the following oscilloscope trace. [The response from the Pico has been amplified for clarity; normally it is the same magnitude as the outgoing signal from the Pi.]

The unusual elements of this protocol make it quite tricky to implement in pure Python, and I must admit the above trace wasn’t generated by my code; it comes from OpenOCD, with an FTDI USB adaptor. The following trace is generated by my code, the frequency is a bit lower (approximately 100 kHz) and is less symmetrical.

The asymmetric waveform isn’t a problem, but a disadvantage of the pure-Python approach is that there are occasions when the CPU is performing other tasks, and it stop driving the SWD interface.

The trace below shows a 300 microsecond pause in the middle of a transfer, and unsurprisingly the Pico doesn’t like this, and returns an error response. Occasional errors like this aren’t a problem, as they are easily handled by the retry mechanism in the PicoReg code.

Error handling

When there is an error, the Pico completely stops communicating over the SWD interface, and ignores all subsequent commands; it is necessary to reset the SWD interface, to re-establish communication.

The reset process is deliberately quite complex, involving the transmission of:

• 8 all-1 bits (FF hex)
• 16 bytes of a specific polynomial
• 4 all-0 bits (0 hex)
• 1 byte to activate the SW-DP interface
• At least 50 all-1 bits

If there are any errors in the process, the Pico will not respond, which makes debugging a bit tricky. Once reset, the connection has to re-established by, transmitting the ID of the core to be accessed; the RP2040 processor has two cores, that are selected using a value of 0x01002927 or 0x11002927. When browsing the peripheral registers, it doesn’t matter which core is selected; the results will be the same.

Higher-level protocol

You might think that, having established contact with the CPU, it would be easy to read the register values, but in reality the interface has several extra levels of complication; some of these I’ve already documented in a previous project, but if you are seeking more information, you really need to read the Arm Debug Interface Architecture Specification (ADI). At the time of writing the latest version (v6.0) is available here.

Copyright (c) Jeremy P Bentham 2021. Please credit this blog if you use the information or software in it.

OpenCV is an incredibly powerful image-processing tool, but it can be difficult to know where to start – how do you grab an image from a camera, and display it in a user-friendly GUI? This post describes such an application, that runs unmodified on a PC or Raspberry Pi, Windows or Linux, Python 2.7 or 3.x, and PyQt v4 or v5.

Installation

On Windows, the OpenCV and PyQt5 libraries can be installed using pip:

pip install numpy opencv-python PyQt5

If pip isn’t available, you should be able to run the module from the command line by invoking Python, e.g. for Python 3:

py -3 -m pip install numpy opencv-python PyQt5

Installing on a Raspberry Pi is potentially a lot more complicated; it is generally recommended to install from source, and for opencv-python, this is a bit convoluted. Fortunately there is a simpler option, if you don’t mind using versions that are a few years old, namely to load the binary image from the standard repository, e.g.

sudo apt update
sudo apt install python3-opencv python3-pyqt5 

At the time of writing, the most recent version of Raspbian Linux is ‘buster’, and that has OpenCV 3.2, which is quite usable. The previous ‘stretch’ distribution has python-opencv version 2.4, which is a bit too old: my code isn’t compatible with it.

With regard to cameras, all the USB Webcams I’ve tried have worked fine on Windows without needing to have any extra driver software installed; they also work on the Raspberry Pi, as well as the standard Pi camera with the ribbon-cable interface.

PyQt main window

Being compatible with PyQt version 4 and 5 requires some boilerplate code to handle the way some functions have been moved between libraries:

import sys, time, threading, cv2
try:
    from PyQt5.QtCore import Qt
    pyqt5 = True
except:
    pyqt5 = False
if pyqt5:
    from PyQt5.QtCore import QTimer, QPoint, pyqtSignal
    from PyQt5.QtWidgets import QApplication, QMainWindow, QTextEdit, QLabel
    from PyQt5.QtWidgets import QWidget, QAction, QVBoxLayout, QHBoxLayout
    from PyQt5.QtGui import QFont, QPainter, QImage, QTextCursor
else:
    from PyQt4.QtCore import Qt, pyqtSignal, QTimer, QPoint
    from PyQt4.QtGui import QApplication, QMainWindow, QTextEdit, QLabel
    from PyQt4.QtGui import QWidget, QAction, QVBoxLayout, QHBoxLayout
    from PyQt4.QtGui import QFont, QPainter, QImage, QTextCursor
try:
    import Queue as Queue
except:
    import queue as Queue

The main window is subclassed from PyQt, with a simple arrangement of a menu bar, video image, and text box:

class MyWindow(QMainWindow):
    text_update = pyqtSignal(str)

    # Create main window
    def __init__(self, parent=None):
        QMainWindow.__init__(self, parent)

        self.central = QWidget(self)
        self.textbox = QTextEdit(self.central)
        self.textbox.setFont(TEXT_FONT)
        self.textbox.setMinimumSize(300, 100)
        self.text_update.connect(self.append_text)
        sys.stdout = self
        print("Camera number %u" % camera_num)
        print("Image size %u x %u" % IMG_SIZE)
        if DISP_SCALE > 1:
            print("Display scale %u:1" % DISP_SCALE)

        self.vlayout = QVBoxLayout()        # Window layout
        self.displays = QHBoxLayout()
        self.disp = ImageWidget(self)    
        self.displays.addWidget(self.disp)
        self.vlayout.addLayout(self.displays)
        self.label = QLabel(self)
        self.vlayout.addWidget(self.label)
        self.vlayout.addWidget(self.textbox)
        self.central.setLayout(self.vlayout)
        self.setCentralWidget(self.central)

        self.mainMenu = self.menuBar()      # Menu bar
        exitAction = QAction('&Exit', self)
        exitAction.setShortcut('Ctrl+Q')
        exitAction.triggered.connect(self.close)
        self.fileMenu = self.mainMenu.addMenu('&File')
        self.fileMenu.addAction(exitAction)

There is a horizontal box layout called ‘displays’, that seems to be unnecessary as it only has one display widget in it. This is intentional, since much of my OpenCV experimentation requires additional displays to show the image processing in action; this can easily be done by creating more ImageWidgets, and adding them to the ‘displays’ layout.

Similarly, there is a redundant QLabel below the displays, which isn’t currently used, but is handy for displaying static text below the images.

Text display

It is convenient to redirect the ‘print’ output to the text box, rather than appearing on the Python console. This is done using the ‘text_update’ signal that was defined above:

    # Handle sys.stdout.write: update text display
    def write(self, text):
        self.text_update.emit(str(text))
    def flush(self):
        pass

    # Append to text display
    def append_text(self, text):
        cur = self.textbox.textCursor()     # Move cursor to end of text
        cur.movePosition(QTextCursor.End) 
        s = str(text)
        while s:
            head,sep,s = s.partition("\n")  # Split line at LF
            cur.insertText(head)            # Insert text at cursor
            if sep:                         # New line if LF
                cur.insertBlock()
        self.textbox.setTextCursor(cur)     # Update visible cursor

The use of a signal means that print() calls can be scattered about the code, without having to worry about which thread they’re in.

Image capture

A separate thread is used to capture the camera images, and put them in a queue to be displayed. The camera may produce images faster than they can be displayed, so it is necessary to check how many images are already in the queue; if more than 1, the new image is discarded. This prevents a buildup of unwanted images.

IMG_SIZE    = 1280,720          # 640,480 or 1280,720 or 1920,1080
CAP_API     = cv2.CAP_ANY       # or cv2.CAP_DSHOW, etc...
EXPOSURE    = 0                 # Non-zero for fixed exposure

# Grab images from the camera (separate thread)
def grab_images(cam_num, queue):
    cap = cv2.VideoCapture(cam_num-1 + CAP_API)
    cap.set(cv2.CAP_PROP_FRAME_WIDTH, IMG_SIZE[0])
    cap.set(cv2.CAP_PROP_FRAME_HEIGHT, IMG_SIZE[1])
    if EXPOSURE:
        cap.set(cv2.CAP_PROP_AUTO_EXPOSURE, 0)
        cap.set(cv2.CAP_PROP_EXPOSURE, EXPOSURE)
    else:
        cap.set(cv2.CAP_PROP_AUTO_EXPOSURE, 1)
    while capturing:
        if cap.grab():
            retval, image = cap.retrieve(0)
            if image is not None and queue.qsize() < 2:
                queue.put(image)
            else:
                time.sleep(DISP_MSEC / 1000.0)
        else:
            print("Error: can't grab camera image")
            break
    cap.release()

The choice of image size will depend on the camera used; all cameras support VGA size (640 x 480 pixels), more modern versions the high-definition standards of 720p (1280 x 720) or 1080p (1920 x 1080).

The camera number refers to the position in the list of cameras collected by the operating system; I’ve defined the first camera as number 1, but the OpenCV call defines the first as 0, so the number has to be adjusted.

The same parameter is also used to define the capture API setting; by default this is ‘any’, which usually works well; my Windows 10 system defaults to the MSMF (Microsoft Media Foundation) backend, while the Raspberry Pi defaults to Video for Linux (V4L). Sometimes you may need to force a particular API to be used, for example, I have a Logitech C270 webcam that works fine on Windows 7, but fails on Windows 10 with an ‘MSMF grab error’. Forcing the software to use the DirectShow API (using the cv2.CAP_DSHOW option) fixes the problem.

If you want to check which backend is being used, try:

print("Backend '%s'" % cap.getBackendName())

Unfortunately this only works on the later revisions of OpenCV.

Manual exposure setting can be a bit hit-and-miss, depending on the camera and API you are using; the default is automatic operation, and setting EXPOSURE non-zero (e.g. to a value of -3) generally works, however it can be difficult to set a webcam back to automatic operation: sometimes I’ve had to use another application to do this. So it is suggested that you keep auto-exposure enabled if possible.

[Supplementary note: it seems that these parameter values aren’t standardised across the backends. For example, the CAP_PROP_AUTO_EXPOSURE value in my source code is correct for the MSMF backend; a value of 1 enables automatic exposure, 0 disables it. However, the V4L backend on the Raspberry Pi uses the opposite values: automatic is 0, and manual is 1. So it looks like my code is incorrect for Linux. I haven’t yet found any detailed documentation for this, so had to fall back on reading the source code, namely the OpenCV videoio ‘cap’ files such as cap_msmf.cpp and cap_v4l.cpp.]

Image display

The camera image is displayed in a custom widget:

# Image widget
class ImageWidget(QWidget):
    def __init__(self, parent=None):
        super(ImageWidget, self).__init__(parent)
        self.image = None

    def setImage(self, image):
        self.image = image
        self.setMinimumSize(image.size())
        self.update()

    def paintEvent(self, event):
        qp = QPainter()
        qp.begin(self)
        if self.image:
            qp.drawImage(QPoint(0, 0), self.image)
        qp.end()

A timer event is used to trigger a scan of the image queue. This contains images in the camera format, which must be converted into the PyQt display format:

DISP_SCALE  = 2                 # Scaling factor for display image

    # Start image capture & display
    def start(self):
        self.timer = QTimer(self)           # Timer to trigger display
        self.timer.timeout.connect(lambda: 
                    self.show_image(image_queue, self.disp, DISP_SCALE))
        self.timer.start(DISP_MSEC)         
        self.capture_thread = threading.Thread(target=grab_images, 
                    args=(camera_num, image_queue))
        self.capture_thread.start()         # Thread to grab images

    # Fetch camera image from queue, and display it
    def show_image(self, imageq, display, scale):
        if not imageq.empty():
            image = imageq.get()
            if image is not None and len(image) > 0:
                img = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
                self.display_image(img, display, scale)

    # Display an image, reduce size if required
    def display_image(self, img, display, scale=1):
        disp_size = img.shape[1]//scale, img.shape[0]//scale
        disp_bpl = disp_size[0] * 3
        if scale > 1:
            img = cv2.resize(img, disp_size, 
                             interpolation=cv2.INTER_CUBIC)
        qimg = QImage(img.data, disp_size[0], disp_size[1], 
                      disp_bpl, IMG_FORMAT)
        display.setImage(qimg)

This demonstrates the power of OpenCV; with one function call we convert the image from BGR to RGB format, then another is used to resize the image using cubic interpolation. Finally a PyQt function is used to convert from OpenCV to PyQt format.

Running the application

Make sure you’re using the Python version that has the OpenCV and PyQt installed, e.g. for the Raspberry Pi:

python3 cam_display.py

There is an optional argument that can be used if there are multiple cameras; the default first camera is number 1.

On Linux, some USB Webcams cause a constant stream of JPEG format errors to be printed on the console, complaining about extraneous bytes in the data. There is some discussion online as to the cause of the error, and the cure seems to involve rebuilding the libraries from source; I’m keen to avoid that, so used the simple workaround of suppressing the errors by redirecting STDERR to null:

python3 cam_display.py 2> /dev/null

Fortunately this workaround is only needed with some USB cameras; the standard Raspberry Pi camera with the CSI ribbon-cable interface works fine.

Source code

Full source code is available here.

For a more significant OpenCV application, take a look at this post.

Copyright (c) Jeremy P Bentham 2019. Please credit this blog if you use the information or software in it.

A basic Web interface has a simple request/response format; the browser requests a Web page, and the server responds with that item. The browser’s request may contain parameters to customise the request, but the requests always come from the browser (i.e. ‘client pull’) rather than the server sending data of its own accord (‘server push’).

As browser applications became more sophisticated, there was a need for general-purpose communication channel between server and browser, so if the server has dynamic data (e.g. constantly fluctuating stock price) it can immediately be ‘pushed’ to the client for display. This is achieved by various extensions to the underlying Web transfer protocol (HTTP), and the latest version of the protocol (HTTP/2) has full support for multiple data streams, but I’ll start by creating a minimal application using a simpler HTTP extension that is compatible with all modern browsers, namely Websockets.

What is a socket?

A socket is a logical endpoint for TCP/IP communications, consisting of an IP address and a port number. On servers, the port number implicitly refers to the service you require from that server; for example, an HTTP Web page request is normally sent to to port 80, or port 443 for the secure version HTTPS.

However, there is no law that says HTTP transactions have to be on port 80; if you are running your own local Web server, you may well have set it up to respond on port 8080, since this is easier than using port 80: port numbers below 1024 are generally for use by the operating system, not user-space programs. You can tell the browser to access a specific port on the server by appending a colon and its number to the Web address.

An additional complication is that communications over the Internet have to get past firewalls, most of which are programmed to block communications on unknown port numbers. For the time being I’ll assume that we are using a private local network, so port 8000 will be fine for the Web server, and port 8001 for the Websocket server. In case you wondered, there is no real rationale behind these numbers; anything above 1023 would do.

Websocket

The protocol starts with a normal HTTP request from browser to Websocket server, but it contains an ‘upgrade’ header to change the connection from HTTP to Websocket (WS). If the server agrees to the change, the connection becomes a transparent data link between client & server, without the usual restrictions on HTTP content.

So the elements we need for a simple demonstration are:

• Web (HTTP) server
• Websocket (WS) server
• Web browser
• Web page with JavaScript code for a Websocket client

This may sound rather complicated, but the reality is really quite easy, as I’ll show below.

Web & Websocket servers

It is tempting to think of combining the Web & Websocket servers into a single entity, but in reality there are two very different requirements; the Web server churns out largely-static pages fetched from disk, while the Websocket server contains application-specific code to organise the flow of non-standard data across the network.

So the solution I’ve adopted is to keep the two servers separate. The simplest possible Web server is included within Python as standard, you just need to run:

# For python 2.7:
  python -m SimpleHTTPServer
# ..or for python3:
  python3 -m http.server

This makes all the files in your current directory visible in the browser, so you can just click on an HTML file to run it. A word of warning: this is can be a major security risk, as an attacker could potentially manipulate the URL to access other information on your system; use with caution.

Next, the Websocket server: there are a few Python libraries containing the protocol negotiation; I’ve chosen SimpleWebSocketServer, which can be installed with ‘pip’ as usual. A minimum of code is needed to make a functioning server (file: websock.py).

# Websocket demo, from iosoft.blog

import signal, sys
from SimpleWebSocketServer import WebSocket, SimpleWebSocketServer

PORTNUM = 8001

# Websocket class to echo received data
class Echo(WebSocket):

    def handleMessage(self):
        print("Echoing '%s'" % self.data)
        self.sendMessage(self.data)

    def handleConnected(self):
        print("Connected")

    def handleClose(self):
        print("Disconnected")

# Handle ctrl-C: close server
def close_server(signal, frame):
    server.close()
    sys.exit()

if __name__ == "__main__":
    print("Websocket server on port %s" % PORTNUM)
    server = SimpleWebSocketServer('', PORTNUM, Echo)
    signal.signal(signal.SIGINT, close_server)
    server.serveforever()

Web page

The browser has a built-in Websocket client, so the Web page just needs to provide:

• Buttons to open & close the Websocket connection
• A display of connection status, and Websocket data
• Some Javascript to link the buttons & display to the Websocket client
• A data source, that will be echoed back by the Python server

Once the Web page has been received and displayed, the user will click a ‘connect’ button to contact the Websocket server. However, the client needs to know the address of the server in order to make the connection; we could just ask the user to fill in a text box with the value, but it is much nicer for the client to work this out, based on the Web server’s address.

Javascript provides a location.host variable that has the current IP address and port number, as shown above.

  // Client for Python SimpleWebsocketServer
  const portnum = 8001;
  var host, server, connected = false;

  // Display the given text
  function display(s)
  {
    document.myform.text.value += s;
    document.myform.text.scrollTop = document.myform.text.scrollHeight;
  }

  // Initialisation
  function init()
  {
    host = location.host ? String(location.host) : "unknown";
    host = host.replace("127.0.0.1", "localhost");
    server = host.replace(/:\d*\b/, ":" + portnum);
    document.myform.text.value = "Host " + host + "\n";
    window.setInterval(timer_tick, 1000);
  }

We use a regular expression to match the Web server port number, and change it to the Websocket server port, on the assumption that the two are hosted at the same IP address. There is also some code to handle the special case of an IP address 127.0.0.1. This address is used by a client, when it is running on the same system as the servers; it should be synonymous with ‘localhost’ but Windows seems to make a distinction between the two, so it is necessary to make a substitution.

Starting and stopping the Websocket connection is relatively straightforward:

  // Open a Websocket connection
  function connect()
  {
    var url = "ws://" + server + "/";
    display("Opening websocket " + url + "\n");
    websock = new WebSocket(url);
    websock.onopen    = function(evt) {sock_open(evt)};
    websock.onclose   = function(evt) {sock_close(evt)};
    websock.onmessage = function(evt) {sock_message(evt)};
    websock.onerror   = function(evt) {sock_error(evt)};
    connected = true;
  }
  // Close a Websocket connection
  function disconnect()
  {
    connected = false;
    websock.close();
  }

Once open, we can send data using a simple function call, and handle incoming data using the callback.

  // Timer tick handler
  function timer_tick()
  {
    if (connected)
      websock.send('*');
  }

  // Display incoming data
  function sock_message(evt)
  {
    display(evt.data);
  }

The resulting display shows the data that has been echoed back by the server:

Web page source

This is the complete source to the Web page (file: websock.html).

<!DOCTYPE html>
<meta charset="utf-8"/>
<title>WebSocket Test</title>
<script language="javascript" type="text/javascript">

  // Client for Python SimpleWebsocketServer
  const portnum = 8001;
  var host, server, connected = false;

  // Display the given text
  function display(s)
  {
    document.myform.text.value += s;
    document.myform.text.scrollTop = document.myform.text.scrollHeight;
  }

  // Initialisation
  function init()
  {
    host = location.host ? String(location.host) : "unknown";
    host = host.replace("127.0.0.1", "localhost");
    server = host.replace(/:\d*\b/, ":" + portnum);
    document.myform.text.value = "Host " + host + "\n";
    window.setInterval(timer_tick, 1000);
  }

  // Open a Websocket connection
  function connect()
  {
    var url = "ws://" + server + "/";
    display("Opening websocket " + url + "\n");
    websock = new WebSocket(url);
    websock.onopen    = function(evt) {sock_open(evt)};
    websock.onclose   = function(evt) {sock_close(evt)};
    websock.onmessage = function(evt) {sock_message(evt)};
    websock.onerror   = function(evt) {sock_error(evt)};
    connected = true;
  }
  // Close a Websocket connection
  function disconnect()
  {
    connected = false;
    websock.close();
  }

  // Timer tick handler
  function timer_tick()
  {
    if (connected)
      websock.send('*');
  }

  // Display incoming data
  function sock_message(evt)
  {
    display(evt.data);
  }

  // Handlers for other Websocket events
  function sock_open(evt)
  {
    display("Connected\n");
  }
  function sock_close(evt)
  {
    display("\nDisconnected\n");
  }
  function sock_error(evt)
  {
    display("Socket error\n");
    websock.close();
  }

  // Do initialisation when page is loaded
  window.addEventListener("load", init, false);

</script>
<form name="myform">
  <h2>Websocket test</h2>
  <p>
  <textarea name="text" rows="10" cols="60">
  </textarea>
  </p>
  <p>
  <input type="button" value="Connect" onClick="connect();">
  <input type="button" value="Disconnect" onClick="disconnect();">
  </p>
</form>
</html> 

Running the demonstration

To run the demonstration, open 2 console windows on the server, and change to a suitable working directory containing the HTML and Python files websock.html and websock.py. In the first window, run the Web server of your choice; you can just run the built-in Python server:

# For python 2.7:
  python -m SimpleHTTPServer
# ..or for python3:
  python3 -m http.server

..but this is relatively insecure, so is only suitable for an isolated private network.

In the second console window, run the ‘websock.py’ application; the console should report:

Websocket server on port 8001

Now run a browser on any convenient system, and enter the address of the server, including the Web server port number after a colon, e.g.

10.1.1.220:8000

You should now see the home page of the Web server; if you are using the built-in Python server, there should be a list of files in the current directory. Click on websock.html, then the connect button; an asterisk should appear every second, having been generated by the Javascript client, and echoed back by the Websocket server. To stop the test, click the disconnect button.

In the next post, I will show how this technique can be expanded to provide a graphical real-time display of server data, watch this space…

Copyright (c) Jeremy P Bentham 2019. Please credit this blog if you use the information or software in it.

In my reporta project, I used a PyQt program to drive an FTDI adapter, producing a graphical display of the CPU’s internals: a real-time animation showing the I/O states, that doesn’t require any additional programming on the target system.

This post aims to produce a more powerful version, namely:

• Use a Raspberry Pi as the interface to the CPU (SWD or JTAG)
• Allow remote diagnosis, by using network communications
• Use standard web browser graphics in place of PyQt

There are many advantages to using the Web browser as a display tool; most importantly, there is no need to install extra software on the display system; you can even use your smartphone as a display device. Wireless communication between the data acquisition & display can be really useful when working on real-world industrial systems, which are often in cramped and inaccessible locations.

First we have to create the display graphic, and I’m using Scalable Vector Graphics (SVG). Since everything is drawn on-the-fly from 2-dimensional x,y positions, it automatically resizes from large to small screens, which is important for mobile devices.

Scalable Vector Graphics

In a previous post, I created some simple graphics in SVG; now I need to draw something that looks like my demonstration target system, with a pushbutton, seven-segment display and ‘blue pill’ STM32F103 CPU module:

My previous PyQt display looked like this:

..but we can do better than that! My first idea was to create the SVG graphics in Inkscape, then add Javascript code to animate them. The problem with this approach is the very large number of tools & settings in Inkscape; it is easy to create something that looks really good visually, but is extremely difficult (or impossible) to animate. So it is much easier to create the SVG from scratch using the Python ‘svgwrite’ library; the display elements can be structured so as to make animation easy.

Background

The background component is a solderless breadboard, with holes at 0.1 inch pitch. This can be created in SVG using a ‘pattern’:

import svgwrite
PIN_PITCH   = 10
PIN_SIZE    = 2
BB_SIZE     = PIN_PITCH*31, PIN_PITCH*11
TILE_SIZE   = PIN_PITCH, PIN_PITCH
TILE_CENTRE = PIN_PITCH/2.0,PIN_PITCH/2.0

# Create maximised SVG drawing
def create_svg(fname, size):
    return svgwrite.Drawing(fname,
            width="100%", height="100%",
            viewBox=("0 0 %u %u" % size),
            debug=False)

# Add a breadboard background pattern
def add_breadboard(dwg, pos=(XPAD,TPAD), size=BB_SIZE):
    dots = svgwrite.pattern.Pattern(width=TILE_SIZE[0], height=TILE_SIZE[1],
                                    id="dots", patternUnits="userSpaceOnUse")
    dots.add(dwg.rect((0,0), TILE_SIZE, fill="#f0f0f0"))
    dots.add(dwg.circle(TILE_CENTRE, 1, fill="white"))
    dwg.defs.add(dots)
    dwg.add(dwg.rect(pos, size, stroke="darkgray", fill="url(#dots)", filter="url(#shadow)"))

dwg = create_svg(FNAME, DWG_SIZE)
add_breadboard(dwg)

A single light grey tile of 10 x 10 units is defined, with a small white dot in the middle. This is used to fill a full-size rectangle; the SVG interpreter automatically duplicates the tile to fill the area.

You may wonder why I have chosen to make the holes 10 units apart, instead of redefining the SVG coordinate system so they are 0.1 units apart, to match the real-world value. The reason is that I’ve found the 10-unit convention to be much more convenient, as it allows positioning to be done with integer values, and the default line width of 1 unit looks fine, so doesn’t need to be modified.

CPU module

A blue rectangle is created, with red ‘pins’ that can be animated to show the I/O on/off status. A list is used to define the pins and their I/O functions:

# STM32F103 'blue pill' board pinout, starting top left
BOARD_PINS=("GND", "GND", "3V3", "NRST","PB11","PB10","PB1", "PB0", "PA7", "PA6",
            "PA5", "PA4", "PA3", "PA2", "PA1", "PA0", "PC15","PC14","PC13","VBAT",
            "PB12","PB13","PB14","PB15","PA8", "PA9", "PA10","PA11","PA12","PA15",
            "PB3", "PB4", "PB5", "PB6", "PB7", "PB8", "PB9", "5V",  "GND", "3V3")

CSS styles are used to define the box colour and pin size. The pin text is also defined, using a ‘writing mode’ of top-to-bottom, which produces the vertical labels.

STYLES = """
    .cpu_style     {stroke:darkblue; stroke-width:1; fill:#b0c0e0}
    .pin_style     {stroke:red; stroke-width:1; fill:red}
    .pin_text      {font-size:6px; writing-mode:tb; font-family:Arial}
"""

dwg.defs.add(dwg.style(STYLES))

It is then just a question of iterating across the pins, drawing them and the optional text labels; these are optional so the same code can be used to draw the (unlabeled) seven-segment display pins.

# Add a dual-in-line part
def add_dil_part(dwg, pos, row_pitch, idents, label=False, style="part_style"):
    g = Group(transform="translate"+str(pos), class_="pin_text")
    row_pins = len(idents) / 2
    g.add(dwg.rect((0,0), (row_pins*PIN_PITCH, row_pitch), class_=style))
    for n, ident in enumerate(idents):
        pos = pin_pos(n, row_pins, PIN_PITCH/2,
                      (PIN_PITCH/2, row_pitch-PIN_PITCH/2))
        g.add(dwg.circle(pos, PIN_SIZE/2, class_="pin_style", id=ident))
        if label:
            pos = pin_pos(n, row_pins, PIN_PITCH/2,
                          (PIN_PITCH, row_pitch-PIN_PITCH*2.5))
            g.add(svgwrite.text.Text(ident, pos))
    dwg.add(g)

add_dil_part(dwg, (100,30), PIN_PITCH*7, BOARD_PINS, True, "cpu_style")

A ‘group’ is used to house the complete part, so it can be styled and positioned as a single item.

Seven-segment display

The same dual-in-line code is used to draw the component base; the pins aren’t actually visible in the real part, but have been included as a handy on/off status indication.

The display segments are drawn using a list of points, arranged so that the line drawing is sequential; this is transformed by the ‘zip’ function into a list of start & end points for each line:

# Dimensions of 7-seg display
D7W,D7H,D7L = 20,20,2       # X and Y seg length, and X-direction lean

# Segment endpoints in the order FABCDEG (for continous drawing)
SEG_LINES   = ((D7L, D7H), (D7L*2,0),(D7L*2+D7W,0),(D7L+D7W,D7H),
               (D7W,D7H*2),(0,D7H*2),(D7L,D7H),    (D7L+D7W,D7H))

# Idents for the display pins, starting top left
DISP_PINS = ("PB11","PB10","GND", "PB1", "PB0",
             "PB12","PB13","GND", "PB14","PB15")

# Idents for the segments, in order ABCDEFGH
SEG_PINS =  ("PB1", "PB0", "PB14","PB13",
             "PB12","PB10","PB11","PB15")

STYLES = """
    .seg_stroke    {stroke:#00a000; stroke-width:5; stroke-linecap:round}
"""

# Add 7 display segments
def add_disp_segs(dwg, pos):
    g = Group(transform="translate"+str(pos), class_="seg_stroke")
    lines = zip(SEG_LINES[:-1], SEG_LINES[1:])
    for n, line in enumerate(lines):
        g.add(dwg.line(*line, id=SEG_PINS[n]))
    dwg.add(g)

Pushbutton

A simple square-plus circle gives an approximation to the real button. The square has slightly rounded corners, using the ‘rx’ parameter.

PB_SIZE = 20

# Add a pushbutton
def add_pb(dwg, pos, ident, size=PB_SIZE, fill="darkred"):
    g = Group(transform="translate"+str(pos))
    g.add(dwg.rect((0,0), (size,size), rx=2, fill=fill, opacity=0.8))
    g.add(dwg.circle((size/2,size/2), size/2, fill=fill, id=ident))
    dwg.add(g)

Drop shadow

Adding a drop shadow to a component is a simple way of creating a 3-dimensional effect.

Confusingly, there are two ways this effect can be achieved; using a CSS definition, or an SVG filter. The CSS method is simpler (since the CSS functionality is a subset of the SVG functionality) but doesn’t work on all browsers, so I’ve used the SVG method instead.

The filter definition consists of a series of steps, with an input and an output; the steps I’ve used are:

• Get the alpha (i.e. the monochrome) values of the image, and offset by 2 units
• Add Gaussian blur to the offset image
• Combine the original image with the offset & blurred image

# Define a shadow filter
def define_shadow(dwg):
    f = dwg.defs.add(dwg.filter(id="shadow", x=0, y=0, width="150%", height="150%"))
    f.feOffset(in_="SourceAlpha", result="AlphaOset", dx="2", dy="2")
    f.feGaussianBlur(in_="AlphaOset", result="AlphaBlur", stdDeviation=2)
    f.feBlend(in_="SourceGraphic", in2="AlphaBlur", mode="normal")

# Add filter to a rectangle, e.g. for the breadboard:
    dwg.add(dwg.rect(pos, size, stroke="darkgray", fill="url(#dots)", filter="url(#shadow)"))

Note the appending of an underscore to ‘in’. This is necessary to avoid a Python syntax error; it is stripped off when the SVG output file is written

Control button, and text display

We need some method of controlling the data connection between the browser and Web server, also displaying the current status. This is achieved by adding an area at the bottom of the graphic.

The ‘connect’ button is drawn as a group, containing a rounded-corner rectangle, and text.

CTRL_SIZE    = 60,19

STYLES = """
    .ctrl_style    {stroke:black; stroke-width:0.5;
                    font-size:9px; font-family:Arial; text-anchor:middle}
"""

# Add a pushbutton control
def add_ctrl_button(dwg, pos, ident, text, onclick, size=CTRL_SIZE, fill="palegreen"):
    g = Group(transform="translate"+str(pos), onclick=onclick, class_="ctrl_style")
    g.add(dwg.rect((0,0), size, rx=5, fill=fill))
    g.add(svgwrite.text.Text(text, (size[0]/2,12), fill="black", id=ident))
    dwg.add(g)

add_ctrl_button(dwg, (20,133), "button1", "Connect", "click_handler()")

The ‘onclick’ parameter will trigger the given JavaScript function when the button is clicked, e.g.

var connected=0;
function click_handler()
{
    if (connected)
        disconnect();
    else
        connect();
}

The status display consists of 2 lines of text; there is no need for scrolling, so the lines are tagged individually:

TEXTBOX_SIZE = 200,20

STYLES = """
    .textbox_style {stroke-width:1.0; stroke:lightgray; fill:none}
    .text_style    {font-size:8px; font-family:Courier}
"""

# Add a text area
def add_textbox(dwg, pos, size=TEXTBOX_SIZE):
    g = Group(transform="translate"+str(pos))
    g.add(dwg.rect((0,0), size, class_="textbox_style"))
    g.add(svgwrite.text.Text("Line1", (5,8), class_="text_style", id="text1"))
    g.add(svgwrite.text.Text("Line2", (5,17), class_="text_style", id="text2"))
    dwg.add(g)

Updating the text in Javascript just requires the ‘textContent’ to be set, e.g.:

// Connect to host
    function connect()
    {
        text2.textContent = "Connected";
        button1.textContent = "Disconnect";
        connected = 1;
    }

    // Disconnect from host
    function disconnect()
    {
        text2.textContent = "Disconnected";
        button1.textContent = "Connect";
        connected = 0;
    }

To be concluded…

The next blog will describe how Raspberry Pi OpenOCD data is used to animate the graphics. It will include a link to the full source code.

Copyright (c) Jeremy P Bentham 2019. Please credit this blog if you use the information or software in it.

Most packages support some form of scripting, so why not program my complete design from scratch, without touching the GUI? FreeCAD is a (free) 3D design package, with a comprehensive Python interface, so seems to be ideal…

This is simple in theory, but a bit tricky in practice; I’ll spare you the many frustrating false-starts I’ve made, and describe some simple ways of producing 3D objects from scratch in Python. This is very much a work-in-progress, but hopefully will provide some useful pointers if you’re a Python programmer doing occasional 3D design.

The examples here have been tested with FreeCAD v0.16, and the current version 0.18

Running Python code

There are several ways of running a Python script in FreeCAD:

1. Entering commands at the console

In the FreeCAD Python console window, try entering:

FreeCAD.newDocument("Unnamed")
import Part
box = Part.makeBox(4, 3, 2)
Part.show(box)

The result is a bit underwhelming; all you can see is the bottom left-hand corner of a square. If you want to see the box in its full 3-D glory, either use the GUI controls to change the viewpoint, or add the following 2 lines:

FreeCADGui.activeDocument().activeView().viewAxonometric()
FreeCADGui.SendMsgToActiveView("ViewFit")

2. Executing as a Macro

Click on Macro then Macros… and you are given a list of Python macro files that can be executed. They are stored in the default location for scripts; you can alter this to a directory of your choosing, by changing the ‘user macros location’ in the dialog box. If you make that change, it is necessary to exit & re-enter FreeCAD for the change to take effect.

If FreeCAD encounters a problem with your script, it will generally give a sensible error message, however very occasionally a script can corrupt the internals of the program, so it fails to respond in the usual manner. Hence, if you are experiencing problems with previously-good code that suddenly doesn’t work, it is worth restarting FreeCAD in case this fixes the problem.

3. Using the FreeCAD editor

You can load a Python script into FreeCAD using the normal GUI File Open. You are then presented with a nice-looking editor window, into which you can paste one of the examples from this blog. When complete, the file can be run by pressing ctrl-F6. Unfortunately, there are some subtle differences when executing a file in this manner, as opposed to the other methods, see my usage of the recompute() function in the later code examples.

4. Executing a file from the FreeCAD Python console

You can directly execute a file by entering a command at the Python console, e.g.

exec(open("/Projects/FreeCAD/test.py").read())

Design methodology

Like many 3D CAD packages, FreeCAD uses the Constructive Solid Geometry (CSG) method, where the final design is built up by adding (fusing) elements together, and subtracting (cutting) one element from another. Simple 3D  objects (cube, cylinder etc.) can be created with a single line of code:

# Simple test of FreeCAD Part scripting, from iosoft.blog

from FreeCAD import Vector
import Part

if FreeCAD.ActiveDocument:
FreeCAD.closeDocument("Unnamed")
doc = FreeCAD.newDocument("Unnamed")
plate = Part.makeBox(50, 30, 2)
verticals = [edge for edge in plate.Edges if edge.BoundBox.ZLength]
plate = plate.makeFillet(5, verticals)
cyl = Part.makeCylinder(10, 2, Vector(20, 15, 0))
plate = plate.cut(cyl)
doc.addObject("Part::Feature", "plate").Shape = plate
FreeCADGui.activeDocument().activeView().viewAxonometric()
FreeCADGui.SendMsgToActiveView("ViewFit")

The result is a plate with rounded corners, and a large off-centre hole:

However the code is more than a single line, so some explanations are in order:

Lines 6-8 remove a previous unnamed document, and create a new one. This means that every time you run the script you get a new clean document to display the result. As a safeguard, if you currently have a named document open, the script will error out

Line 9 creates a square plate, size 50 x 30 units, and 2 units thick.

Lines 10 & 11 create the rounded corners (‘fillets’ in FreeCAD terminology), with a radius of 5 units. The makeFillet method requires a list of edges, and we only want to modify the vertical edges, so the list of edges is filtered by checking the z-dimension length is non-zero.

Lines 12 & 13 create a cylinder with 10 units radius and 2 units high, offset from the origin using a ‘Vector’ object, which defines a position in 3-dimensional (x,y,z) space. The plate is then cut with the cylinder, creating a hole 20 units in diameter.

Line 14 adds the resulting object to the current document; without this step, the object won’t be visible.

Lines 15 & 16 set a perspective view, and adjust the zoom level so the part fits in the display space.

The ‘labels and attributes’ on the left-hand side of the screen show there is only one defined object, named ‘plate’; if you click on that you can modify its placement (i.e. its position and orientation) but none of the other design parameters.

Workbenches

FreeCAD has various sets of software tools, divided up into ‘workbenches’. This division isn’t just for the GUI, it also applies to scripting; for example, the above Python code used tools imported from the Part workbench. There is some overlap between workbenches, so there can be other ways of creating the same object, for example starting with 2-dimensional sketches in the Draft workbench:

# Simple test of FreeCAD Draft scripting, from iosoft.blog

from FreeCAD import Vector
import Draft

if FreeCAD.ActiveDocument:
FreeCAD.closeDocument("Unnamed")
doc = FreeCAD.newDocument("Unnamed")

rect = Draft.makeRectangle(50, 30, face=True)
rect.FilletRadius = 5
rect.ViewObject.Visibility = False
plate = doc.addObject("Part::Extrusion", "plate")
plate.Base, plate.Dir = rect, Vector(0, 0, 2)

circ = Draft.makeCircle(10, face=True)
rotation = App.Rotation(Vector(0,0,0), 0)
circ.Placement = App.Placement(Vector(20, 15, 0), rotation)
circ.ViewObject.Visibility = False
cyl = doc.addObject("Part::Extrusion", "cylinder")
cyl.Base, cyl.Dir = circ, Vector(0, 0, 2)

FreeCADGui.activeDocument().activeView().viewAxonometric()
FreeCADGui.SendMsgToActiveView("ViewFit")

cutplate = doc.addObject("Part::Cut", "cutplate")
cutplate.Base, cutplate.Tool = plate, cyl

doc.recompute()

To turn the 2-dimensional x-y sketches into 3-dimensional objects, they are extruded in the z-plane. The Draft workbench has no extrusion capability, so this is done by adding extrusion objects to the document. It is important to set the original 2-dimensional sketches as transparent (set visibility false), otherwise they will form a thin layer that obscures the cutout.

I must explain the last 5 lines of code, where I set the viewing mode, cut the plate with the cylinder, then do a recompute. This is a workaround for some minor issues I found in the current FreeCAD versions, which may well be fixed by now:

• If the view mode is set after doing the cut, you don’t get the correct perspective view.
• If the final recompute is omitted, everything works fine when executed as a standalone script, but when the same script is executed from the edit screen, the document is blank; none of the objects are visible.

Apart from these issues, the graphical end result looks exactly the same as with the Part workbench, but note the collection of nested objects in the left-hand window. This is CSG in action; the cut plate is derived from a plate and a cylinder, which in turn are derived from a rectangle and circle.

An advantage of this object hierarchy is that the design parameters can be changed within the GUI; for example, try changing the circle radius in the attributes window from 10 to 14 units, hit F5 to recompute, and the hole diameter will increase to 28 units. The primary focus of this blog is to use scripting rather than the GUI, but it can be useful to manually change a few parameters, and instantly see the result.

An interesting quirk of this hierarchy is that the cut plate has effectively taken possession of the circle and cylinder, since they are its sub-objects. This raises the question: what happens if we cut 2 plates with the same cylinder, which of the plates will own the cylinder? Let’s try it out:

# Simple test of FreeCAD CSG hierarchy, from iosoft.blog

from FreeCAD import Vector
import Draft

if FreeCAD.ActiveDocument:
FreeCAD.closeDocument("Unnamed")
doc = FreeCAD.newDocument("Unnamed")

rect = Draft.makeRectangle(50, 30, face=True)
rect.FilletRadius = 5
rect.ViewObject.Visibility = False
plate = doc.addObject("Part::Extrusion", "plate")
plate.Base, plate.Dir = rect, Vector(0, 0, 2)

plate2 = doc.addObject("Part::Extrusion", "plate2")
plate2.Base, plate2.Dir = rect, Vector(0, 0, 2)
rotation = App.Rotation(Vector(0,0,0), 0)
plate2.Placement = App.Placement(Vector(0, 0, 10), rotation)

circ = Draft.makeCircle(10, face=True)
circ.Placement = App.Placement(Vector(20, 15, 0), rotation)
circ.ViewObject.Visibility = False
cyl = doc.addObject("Part::Extrusion", "cylinder")
cyl.Base, cyl.Dir = circ, Vector(0, 0, 12)

FreeCADGui.activeDocument().activeView().viewAxonometric()
FreeCADGui.SendMsgToActiveView("ViewFit")

cutplate = doc.addObject("Part::Cut", "cutplate")
cutplate.Base, cutplate.Tool = plate, cyl

cutplate2 = doc.addObject("Part::Cut", "cutplate2")
cutplate2.Base, cutplate2.Tool = plate2, cyl

doc.recompute()

..and the answer to the question is..

FreeCAD v0.18 has duplicated the single cylinder, so it appears twice in the hierarchy. Even though they appear to be separate, these 2 circles & cylinders are actually linked; if you change the radius of one circle, both hole sizes change. The trick is to note that the duplicate objects have the same name; if they were separate items, FreeCAD would have automatically renamed the second one by adding a numeric suffix.

FreeCAD v0.16 does not duplicate the parts, the circle & cylinder only appear once under cutplate2, so caution is needed when working with the older version.

Copyright (c) Jeremy P Bentham 2019. Please credit this blog if you use the information or software in it.

I do a lot of work with serial comms; TeraTerm is an excellent serial terminal, but sometimes a customised application is required, for example when dealing with binary data.

The following blog describes a simple Windows & Linux serial terminal, that can be adapted to handle special protocols; it also demonstrates the creation of Python threads, and can serve as a basis for other multi-threaded applications.

It won’t win any awards for style, but could come in handy next time you encounter an obscure serial protocol.

Compatibility

My code is compatible with Python 2.7 and 3.x, PyQt v4 and v5, running on Windows or Linux. This necessitates some rather clunky inclusions at the top of the file:

try:
    from PyQt4 import QtGui, QtCore
    from PyQt4.QtGui import QTextEdit, QWidget, QApplication, QVBoxLayout
except:
    from PyQt5 import QtGui, QtCore
    from PyQt5.QtWidgets import QTextEdit, QWidget, QApplication, QVBoxLayout
try:
    import Queue
except:
    import queue as Queue

There is also an issue with the different ways Python 2 and 3 handle serial data; older versions assume that the data type is an ASCII string, while in later versions it is a binary type.

This can lead to all sorts of problems (such as unwanted exceptions) so I’ve included specific functions to ensure that all outgoing data is converted from the internal representation (which could be Unicode) to a string of bytes, and incoming data is converted from the external type (data bytes or string) to a string type:

# Convert a string to bytes (for Python 3)
def str_bytes(s):
    return s.encode('latin-1')

# Convert bytes to string (if Python 3)
def bytes_str(d):
    return d if type(d) is str else "".join([chr(b) for b in d])

You’ll need to install pySerial by the usual methods (e.g. ‘pip install’).

Using threading

The key problem with serial communications is the time it takes; the slower the data rate, the longer the delay before anything happens. Without threading (or any other mitigation scheme) the User Interface (UI) will lock up after each command, waiting for the response. At best, this makes the application appear sluggish and unresponsive; at worst, it can appear to have failed, waiting for a response that never comes.

Threading allows the UI to carry on interacting with the user, while simultaneously keeping the serial link alive. Creating a new thread is really easy in Python; you just subclass QThread, and instantiate it:

class SerialThread(QtCore.QThread):
    def __init__(self, portname, baudrate):
        QtCore.QThread.__init__(self)
        self.portname, self.baudrate = portname, baudrate
...
class MyWidget(QWidget):
    ...
    self.serth = SerialThread(portname, baudrate)

I have chosen to supply the serial parameters (port name & baud rate) when the thread object is instantiated, so they are available when the thread is started; this is done by calling the start() method, which will call a run() method in the QThread class:

# In serial thread:
def run(self):
    print("Opening %s at %u baud" % (self.portname, self.baudrate))
...
# In UI thread:
    self.serth.start()

The run() method needs to keep running in a perpetual loop, but it must be possible to terminate that loop when the program exits – Python won’t do it automatically. I use a global variable, that can be set false to terminate, e.g. in pseudocode:

def run(self):
    [starting: open serial port]
    while self.running:
        [check for incoming characters]
        [check for outgoing characters]
    [finished: close serial port]

When the application is closing, it terminates the thread by setting the boolean variable false, then (very importantly) waits for the thread to finish its execution:

def closeEvent(self, event):
    self.serth.running = False
    self.serth.wait()

Transmitting

The user will be entering keystrokes in the UI thread, and it is tempting to call the serial transmit function from that thread, but this isn’t a good idea; it is better to pass the keystrokes across to the serial thread for transmission, and we need a thread-safe method of doing this. That means we can’t just use a global shared string variable; Python does a lot of behind-the-scenes processing that could lead to an unpredictable result. Instead, we’ll use a First In First Out (FIFO) queue:

# In UI thread..
txq = Queue.Queue()
...
txq.put(s)           # Add string to queue
...
# In serial thread..
if not txq.empty():
    txd = txq.get()  # Get string from queue

So the serial thread polls the transmit queue for any data, outputting it to the serial port.

Receiving

We could use the same technique for received data; the serial thread could add it to a queue that is polled by the UI thread, and somehow trigger a UI redraw when the new data arrives, but I prefer to use a signal; the data is attached to that signal, and is received by a UI function that has registered a connection. The signal has to be in a class definition, and must specify the type of data that will be attached:

class MyWidget(QWidget):
    text_update = QtCore.pyqtSignal(str)

The signal is connected to a function that will process the data:

self.text_update.connect(self.append_text)

So now the serial thread just has to generate a signal when new data is received:

self.text_update.emit(text)

This technique would be quite adequate, but I do like having the output from all my ‘print’ function calls redirected to the same window; it makes for cleaner error reporting when things go wrong, rather than having a separate console with error messages. This is done by redirecting stdout to my widget, and adding write() and flush() handlers:

class MyWidget(QWidget):
    text_update = QtCore.pyqtSignal(str)
    def __init__(self, *args): 
        ...
        self.text_update.connect(self.append_text)
        sys.stdout = self
        ...
    def write(self, text):
        self.text_update.emit(text)  
    def flush(self):
        pass
        ...
    def append_text(self, text):        
        [add text to UI display]

So now, every time I make a print() call, a signal is sent to my append_text function, where the display is updated. The use of a signal means that I can still call print() from any thread, without fear of strange cross-threading problems.

Polling in serial thread

The serial thread is just polling for incoming and outgoing characters, and if there are none, the processor will execute the ‘while’ loop really quickly. In the absence of any delays, it will consume a lot of CPU time just checking for things that don’t exist. This may appear harmless, but it is quite alarming for the user if the CPU fan starts spinning rapidly whenever your application is running, and a laptop user won’t be happy if you needlessly drain their battery by performing pointless tasks. So we need to add some harmless time-wasting to the polling loop, by frequently returning control to the operating system. This can be done by calling the ‘sleep’ function, but we still want the software to be responsive when some serial data actually arrives. A suitable compromise is to use a serial read-function with a timeout, so the software ‘blocks’ (i.e. stalls) until either some characters are received, or there is a timeout:

self.ser = serial.Serial(self.portname, self.baudrate, timeout=SER_TIMEOUT)
...
s = self.ser.read(self.ser.in_waiting or 1)

In case you are unfamiliar with the usage, the ‘or’ function returns the left-hand side if it is true (non-zero), otherwise the right-hand side. So every read attempt is at least 1 character, or more characters if they are available. If none are present, the read function waits until the timeout, so when the serial line is idle, most time will be spent waiting in the operating system.

User Interface

This has been kept as simple as possible, with just a text box as a main widget. One minor complication is that as standard, the text box (which is actually a QTextEdit control) will capture and display all keystrokes, so we need to subclass it to intercept the keys, and call a handler function that adds them to the serial transmit queue. I didn’t want to burden the text box with this functionality, so put the handler in its parent, which is the main widget:

class MyTextBox(QTextEdit):
    def __init__(self, *args): 
        QTextEdit.__init__(self, *args)

    def keyPressEvent(self, event):
        self.parent().keypress_handler(event)

The keystroke handler in the main widget gets the character from the key event, and checks for a ctrl-V ‘paste’ request; I’ve included this feature because I find it useful to cut-and-paste frequently-used serial commands from a document, rather than re-typing them every time.

# In main widget:
def keypress_handler(self, event):
    k = event.key()
    s = RETURN_CHAR if k==QtCore.Qt.Key_Return else event.text()
    if len(s)>0 and s[0]==PASTE_CHAR:
        cb = QApplication.clipboard() 
        self.serth.ser_out(cb.text())
    else:
        self.serth.ser_out(s)

Interestingly, with PyQt5 you can cut-and-paste full 8-bit data (i.e. bytes with the high bit set), but this doesn’t seem to work in PyQt4, which only accepts the usual ASCII character set.

I haven’t included any menu options, you have to specify the COM port on the command-line using the -c option, and baud rate using -b. There is also a -x option to display incoming data in hexadecimal, for example:

Windows:
  python pyqt_serialterm.py -c com2 -b 9600 -x
Linux:
  python pyqt_serialterm.py -c /dev/ttyUSB0 -b 115200

On Linux, if you want non-root access to a serial port, you will generally need to add your username to the ‘dialout’ group:

sudo usermod -a -G dialout $USER

..then log out & back in.

Source code

The source code is posted here.

Copyright (c) Jeremy P Bentham 2019. Please credit this blog if you use the information or software in it.

My previous attempt at real-time graphics involved direct-drawing the shapes using Python and PyQt. This works well for a standalone system, but is less useful in the current world of ‘everything-on-a-browser’. To make the graphics Web-friendly, there is a choice; either create an animated bitmap (e.g. GIF, PNG or JPEG) or use vector graphics (SVG).

Vector graphics have various advantages; not just that they can be resized without degradation (pixellation) but also that they can preserve some of the structural characteristics of the image that is being animated. For example, we can draw a detailed circuit diagram complete with voltage and current measurements, and if it is too large to display in a browser screen, the user can zoom out to get the big picture, then zoom in to see the data in a specific area; the components will always appear well-drawn, no matter what scale is used.

If the components are drawn sensibly they can easily be annotated & animated to reflect the current data values. Unfortunately, although most graphics packages can export SVG, very few preserve the underlying structure of the parts. For example, the above circuit diagram was originally exported from Visio as an 800K file, filled with tiny vector & text fragments, that were very difficult to relate back to the original components. The final version, created by a Python program from a netlist, was 33KB in size, and much easier to animate.

I certainly wouldn’t give up on the idea of animating pre-existing circuit diagrams, but for the purposes of this blog, am taking the easy way out, and creating the graphics from scratch in Python. Also, it is quite a big step to create & animate a full circuit diagram in one go, so I’ll start with something much simpler, a 7-segment digital clock.

This will involve Python, SVG, style sheets and even a smattering of Javascript, so is complicated enough for starters. If you are going to do a lot of graphical manipulation, it is well worth reading the SVG specification.

svgwrite

My code is compatible with Python 2.7 or 3.x, you just need to install the ‘svgwrite’ package using pip or pip3.

As its name suggests, svgwrite is a relatively slim wrapper that simplifies the task of writing SVG files, for example:

# Create simple SVG
import svgwrite
FILENAME      = "simple.svg"
WIDTH, HEIGHT = 250, 150

dwg = svgwrite.Drawing(FILENAME, (WIDTH, HEIGHT))
dwg.add(dwg.rect((0,0),
                 (WIDTH-1,HEIGHT-1),
                 stroke="red",
                 fill="none"))
dwg.add(dwg.line((0,0),
                 (WIDTH-1,HEIGHT-1),
                 stroke="green",
                 stroke_width=8))
dwg.add(dwg.circle((WIDTH/2,HEIGHT/2),
                   HEIGHT/2,
                   stroke_width=3,
                   stroke="salmon",
                   fill="moccasin"))
dwg.add(dwg.circle((WIDTH/2,HEIGHT/2),
                   HEIGHT/3,
                   stroke_width=3,
                   stroke="salmon",
                   fill="moccasin"))
dwg.save()

If the resulting SVG file is dragged into a browser, it looks like:

There are various points to note (apart from the author’s strange colour choices):

• The origin (x=0, y=0) is in the top left corner
• Coordinates are specified as (x,y) tuples
• The order of drawing is significant: earlier objects may be obscured by later
• Lines outside the drawing area are clipped
• To eliminate fill, you have to use ‘fill=”none”‘, rather than the more Pythonic ‘fill=None’

It is instructive to view the resulting file in a text editor; after the namespace boilerplate, there is a close match with the Python code:

<?xml version="1.0" encoding="utf-8" ?>
<svg baseProfile="full" height="150" version="1.1" width="250" 
 xmlns="http://www.w3.org/2000/svg" 
 xmlns:ev="http://www.w3.org/2001/xml-events"
 xmlns:xlink="http://www.w3.org/1999/xlink">
 <defs />
 <rect fill="none" height="149" stroke="red" width="249" x="0" y="0" />
 <line stroke="green" stroke-width="8" x1="0" x2="249" y1="0" y2="149" />
 <circle cx="125" cy="75" fill="moccasin" r="75" stroke="salmon" stroke-width="3" />
 <circle cx="125" cy="75" fill="moccasin" r="50" stroke="salmon" stroke-width="3" />
</svg>

Style

The most obvious issue is the repetition of the colour definitions; using a style sheet would shorten the file, and separate the style from the drawing commands, making it easier to change the colours, for example:

import svgwrite

FILENAME      = "simple.svg"
WIDTH, HEIGHT = 250, 150

STYLES = """
  .circle_style  {stroke-width:3; stroke:salmon; fill:moccasin}
"""

dwg = svgwrite.Drawing(FILENAME, (WIDTH, HEIGHT))
dwg.defs.add(dwg.style(STYLES))
dwg.add(dwg.rect((0,0),
                 (WIDTH-1,HEIGHT-1),
                 stroke="red",
                 fill="none"))
dwg.add(dwg.line((0,0),
                 (WIDTH-1,HEIGHT-1),
                 stroke="green",
                 stroke_width=8))
dwg.add(dwg.circle((WIDTH/2,HEIGHT/2),
                   HEIGHT/2,
                   class_="circle_style"))
dwg.add(dwg.circle((WIDTH/2,HEIGHT/2),
                   HEIGHT/3,
                   class_="circle_style"))
dwg.save(pretty=True)

The browser display is exactly the same, and the SVG code now has an entry in the ‘defs’ section, with a strange addition:

<![CDATA[
  .circle_style {stroke-width:3; stroke:salmon; fill:moccasin}
]]></style>

The CDATA encapsulation is a signal to the browser that the normal search for HTML tags should be disabled; it isn’t really necessary in this case, but is essential if, for example, you are inserting a Javascript comparison such as ‘a < b’, since the browser would normally treat ‘<‘ as the start of an HTML tag.

Two important points:

1. Note the underscore on the end of ‘class_’ in the Python code. If you forget that, Python will error out, as ‘class’ is a reserved word; svgwrite automatically strips off the trailing underscore when writing the SVG file.
2. If you make any errors in the definition, the whole class will be ignored, and the object will be drawn in the default style.

The first issue is very common; you will see many complaints from svgwrite users that they can’t assign a class. The second issue can be annoying, since a trivial mistake can produce an ugly (or invisible!) mess, for example the following display results from putting the colour names in quotes (stroke:”salmon”; fill:”moccasin”).

The ‘developer tools’ mode in Chrome has highlighted the incorrect colour definitions.

Grouping

A useful SVG feature is that several items can be grouped together and treated as one, for example, instead of adding the 2 circles directly to the drawing, they can be added to a group, which is then added to the drawing:

g = svgwrite.container.Group(class_="circle_style")
g.add(dwg.circle((WIDTH/2,HEIGHT/2), HEIGHT/2))
g.add(dwg.circle((WIDTH/2,HEIGHT/2), HEIGHT/3))
dwg.add(g)

This is of most use when handling a complex shape, as all grouped items will be moved as one; if you want to move the 2 grouped circles to the right by 10 pixels, the first line becomes:

g = svgwrite.container.Group(class_="circle_style", transform="translate(10 0)")

You might think that the ‘add’ function would allow an offset to be applied to the items, but that isn’t true; it simply inserts the item into the file.

Using predefined items

If the the same element appears 2 or more times in your graphics, it is worthwhile defining it in the ‘defs’ section, then just adding it to the drawing with a ‘use’ command, which also allows the predefined graphic to be positioned and styled as required.

g = svgwrite.container.Group()
g.add(dwg.circle((WIDTH/2,HEIGHT/2), HEIGHT/2))
g.add(dwg.circle((WIDTH/2,HEIGHT/2), HEIGHT/3))
dwg.defs.add(g)
dwg.add(dwg.use(g, class_="circle_style"))
dwg.add(dwg.use(g, insert=(40,0), style="stroke:green; fill:palegreen"))
dwg.save()

You’ll note that I’ve removed the class from the group definition, and instead applied class & style definitions when the graphic is used. This isn’t compulsory; you can specify a class when the group is defined, but then it remains fixed, which limits the flexibility of your predefined objects – once a graphic is defined you can’t change its underlying structure.

It can also be really useful to name the predefined items using an ‘id’ parameter; they can then be inserted into the document by referencing that name, improving readability.

g = svgwrite.container.Group(id="circles")
g.add(dwg.circle((WIDTH/2,HEIGHT/2), HEIGHT/2))
g.add(dwg.circle((WIDTH/2,HEIGHT/2), HEIGHT/3))
dwg.defs.add(g)
dwg.add(dwg.use("#circles", class_="circle_style"))

The ‘#’ prefix is necessary to turn the ID into an Internationalised Resource Identifier (IRI).

Colour gradient

The green line looks a bit boring, so I wanted to add a colour gradient, making it look 3-dimensional. All the examples I’ve seen are of horizontal or vertical boxes, but I needed the shading to be aligned along the axis of the diagonal line, so it looks like a round bar. My intention was to create some very clever code to do this, but the result always looked terrible, so in the end I just hacked the parameters to make it look roughly right. Sorry!

lg = svgwrite.gradients.LinearGradient(
     start=(0,0), end=(40,0),
     id="blue_grad", gradientUnits="userSpaceOnUse",
     gradientTransform="rotate(120)")
lg.add_colors(['blue','white'])
dwg.defs.add(lg)
dwg.add(dwg.line(
    (0,0), (WIDTH-1,HEIGHT-1),
    stroke="url(#blue_grad)",
    stroke_width=50))
dwg.save()

Modifying objects

In the above code, we’ve been creating a graphic object and and adding it to the drawing in a single line. Alternatively, we can access and modify the object’s attributes using its dictionary, for example changing the stroke colour:

line = dwg.line((10,0), (110, 100), stroke="red")
...
# Change to green using either..
line["stroke"] = "green"
# ..or..
line.update({"stroke":"green"})
...
dwg.add(line)

Javascript

The intention is to create a working seven-segment clock display; we could possibly do this in Python by continually updating an SVG file, and persuading the browser to reload it, but a much simpler method is to use Javascript.

Each object (or group of objects) to be animated must have a unique ‘id’ tag.

dwg.add(dwg.line((20,0), (120, 100), stroke="red", id="myline"))

To change this line, we have to wait until the browser has loaded the graphics, so the code is triggered by window.onload event:

SCRIPT = """
window.onload = function()
{
    var myline = document.getElementById("myline");
    if (myline)
        myline.setAttribute("stroke", "yellow");
}
"""
dwg.defs.add(dwg.script(content=SCRIPT))

That’s how easy it is to modify a graphic at run-time using Javascript.

If you are a newcomer to that language, the developer’s console in the browser is incredibly useful; you can add diagnostic printouts using console.log(), and they will only show up when the developer tools are enabled. Here is the Chrome console when ‘console.log(myline);’ has been added to Javascript; hovering the cursor over the text display causes the associated graphic bounding box to be highlighted:

SVG clock

I started off by drawing the segments using simple flat-colour lines:

However the colour gradients look much more attractive..

..so I decided to use them instead. This triggered a cascade of problems, because using colour gradients on lines is much more restrictive than rectangles, in respect of the overall dimensions the browser uses to calculate gradient values (see the documentation on gradientUnits); when lines are drawn in various on-screen positions using the same gradient, only one is the correct colour.

The way round this problem was to draw only one shaded line, then use the ‘transform’ function to rotate and move a copy of that line. The parameters for the transformation are in the form of a 6-value array, which allows complex changes to be made in one step. Simplistically, the following transformations can be made:

(1, 0, 0, 1, x, y)          Move (translate) by x,y
(sx, 0, 0, sy, 0, 0)        Scale by sx, sy
(cos, sin, -sin, cos, 0, 0) Rotate by angle
(1, 0, tan, 1, 0, 0)        x-skew by angle
(1, tan, 0, 1, 0, 0)        y-skew by angle

A transformation array is needed for each of the 7 segments, so if segment A is the original drawn line:

..the 7 segment transformations are..

# Transform matrix for each segment
SEG_MATRIX = ((1, 0,  0, 1,      0,        0), # A
              (0, 1, -1, 0, SEGLEN,        0), # B
              (0, 1, -1, 0, SEGLEN,   SEGLEN), # C
              (1, 0,  0, 1,      0, SEGLEN*2), # D
              (0, 1, -1, 0,      0,   SEGLEN), # E
              (0, 1, -1, 0,      0,        0), # F
              (1, 0,  0, 1,      0,   SEGLEN)) # G

The resulting lines are in a rectangular grid; the final flourish is to use a ‘skew’ transformation on the complete digit to give it a lean to the right:

# Matrix to slightly skew the displays
SKEW_MATRIX = "matrix(1,0,-0.1,1,0,0)"
...
g = Group(transform=SKEW_MATRIX)

The remaining question is how to do the animation; the simplest (and hopefully fastest) method I could think of is to stack all the digits 0 – 9 on top of each other, setting them transparent (opacity = 0). Each one is labelled with its position on the display, and its value, so ‘digit00’ is the left-most digit, with a value of 0, ‘digit01’ is the left-most digit with a value of 1, ‘digit10’ is the second digit with a value of 0, and so on. All the Javascript code has to do is set the opacity on the required digits, so to display the number 2345, digit02, digit13, digit24, and digit35 are opaque, the rest are transparent.

Source code

Here is the entire source code, compatible with Python 2.7 or 3.x

To view the resulting SVG file, click here.

# SVG clock with 7-segment display

import svgwrite
from svgwrite.container import Group

# 7-seg display
SEGWID      = 5             # Width & length of a segment
SEGLEN      = 20
NDIGITS     = 6             # Number of digits
DIG_PITCH   = 35            # Pitch of digits
DIG_SCALE   = 1             # Scaling factor for main digits
FRAC_SCALE  = 0.7           # Scaling factor for fractional part
LMARGIN     = 20            # Left margin
VMARGIN     = 5             # Vertical margins

GRAD_COLOURS= 'lightblue','blue'# Gradient colours
FNAME       = "svg_clock.svg"   # Filename

DWG_SIZE    = DIG_PITCH*NDIGITS, (VMARGIN+SEGLEN)*2

# Transform matrix for each segment
SEG_MATRIX = ((1,0, 0,1,0,      0),        # A
              (0,1,-1,0,SEGLEN, 0),        # B
              (0,1,-1,0,SEGLEN, SEGLEN),   # C
              (1,0, 0,1,0,      SEGLEN*2), # D
              (0,1,-1,0,0,      SEGLEN),   # E
              (0,1,-1,0,0,      0),        # F
              (1,0, 0,1,0,      SEGLEN))   # G

# Matrix to slightly skew the displays
SKEW_MATRIX = "matrix(1,0,-0.1,1,0,0)"

# Matrix to scale a segment down to a decimal point
DP_MATRIX   = "matrix(%g,0,0,1,0,0)" % (float(SEGWID)/SEGLEN)

# Segments to be activated for digits 0 - 9
dig_segs = ("abcdef", "bc", "abdeg", "abcdg", "bcfg",
            "acdfg", "acdefg", "abc", "abcdefg", "abcdfg")

SCRIPT = """
    var ndigits=6, last;

    // Initialise
    window.onload = function()
    {
        last = 0;
        setInterval(tick_handler, 100)
    }

    // Timer tick handler
    function tick_handler()
    {
        var now = new Date();
        var t = now.getHours()*10000 + now.getMinutes()*100 +
                now.getSeconds();
        if (t != last)
            set_value(t, last);
        last = t;
    }

    // Set a value on the display
    function set_value(val, oldval)
    {
        var i;
        for (i=0; i<ndigits; i++)
        {
            set_digit(ndigits-1-i, oldval, 0);
            set_digit(ndigits-1-i, val, 1);
            val = Math.floor(val / 10);
            oldval = Math.floor(oldval / 10);
        }
    }

    // Set opacity of a single display digit
    function set_digit(dig, val, op)
    {
        var digit = document.getElementById("digit"+dig%10+val%10);
        if (digit)
            digit.setAttribute("opacity", op);
    }
"""

# Create maximised SVG drawing
def create_svg(fname, size):
    return svgwrite.Drawing(fname,
            width="100%", height="100%",
            viewBox=("0 0 %u %u" % size),
            debug=False)

# Add Javascript to drawing
def add_script(dwg, script):
    dwg.defs.add(dwg.script(content=script))

# Create linear gradient, add to drawing defs
def create_lin_grad(dwg, id, start, end, colours):
    grad = svgwrite.gradients.LinearGradient(
           start=start, end=end,
           id=id, gradientUnits="userSpaceOnUse")
    grad.add_colors(colours)
    dwg.defs.add(grad)

# Create a single segment, add to drawing defs
def create_seg(dwg, id, length, width, stroke):
    seg = dwg.line((0,0), (length,0), id=id,
                   stroke="url(#%s)" % stroke,
                   stroke_width=width, stroke_linecap="round")
    dwg.defs.add(seg)

# Create digits 0 - 9 by transforming a segment, add to drawing defs
def create_digits(dwg, digid, segid):
    for dig, segs in enumerate(dig_segs):
        g = Group(id="%s%u" % (digid,dig), transform=SKEW_MATRIX)
        for seg in segs:
            num = ord(seg) - ord('a')
            mat = SEG_MATRIX[num]
            g.add(dwg.use('#'+segid, transform="matrix%s" % str(mat)))
        dwg.defs.add(g)

# Create decimal point by scaling a segment, add to drawing defs
def create_dp(dwg, dpid, segid):
    g = Group(id=dpid, transform=SKEW_MATRIX)
    g.add(dwg.use('#'+segid, transform=DP_MATRIX))
    dwg.defs.add(g)

# Add seven-segment display
# Each digit is a group of coincident transparent chars 0 - 9
def add_display(dwg, pos, pitch):
    dpos = list(pos)
    for n in range(0, NDIGITS):
        scale = DIG_SCALE if n<4 else FRAC_SCALE
        g = Group(transform = "translate(%g %g) scale(%g)" %
            (dpos[0], dpos[1], scale))
        for i in range(0, 10):
            g.add(dwg.use("#dig%u"%i, id="digit%u%u"%(n,i), opacity=0))
        dwg.add(g)
        if n == 2:
            dwg.add(dwg.use("#dp", insert=(dpos[0]-SEGLEN*0.7,
                                           dpos[1]+SEGLEN*2)))
        dpos[0] += pitch * scale

if __name__ == '__main__':
    dwg = create_svg(FNAME, DWG_SIZE)
    add_script(dwg, SCRIPT)
    create_lin_grad(dwg, "blue_grad", (0,0), (0,SEGWID*3/4),
                    GRAD_COLOURS)
    create_seg(dwg, "seg", SEGLEN, SEGWID, "blue_grad")
    create_digits(dwg, "dig", "seg")
    create_dp(dwg, "dp", "seg")
    add_display(dwg, (LMARGIN, VMARGIN), DIG_PITCH)
    dwg.add(dwg.text("iosoft.blog",
            (LMARGIN-2+DIG_PITCH*4, DWG_SIZE[1]-5),
            style="font-size:8px; font-family:Arial", fill="gray"))
    dwg.save(pretty=True)

# EOF

Copyright (c) Jeremy P Bentham 2019. Please credit this blog if you use the information or software in it.

In previous blog posts, I have described how an FTDI USB device can be programmed in Python to access the SWD bus of an ARM microprocessor. This allows the internals of the CPU to be accessed, without disrupting the currently running program.

In this blog I take the process one step further, and add a graphical front-end, that shows the CPU activity in real time; if you want to see it in action, take a look at this video.

If you need a more powerful debug system, take a look at my post OpenOCD on the Raspberry Pi. I’ve also created a remote graphical front-end that uses a Web browser for display, instead of running PyQt locally, click here for details.

Hardware

The target system in the demonstration is a ‘blue pill’ STM32F103 board, with a 7-segment display and pushbutton. This CPU board is particularly convenient because it has a 4-pin connector with SWD clock & data; it can be seen on the right-hand side of the photograph above.

The SWD connection is described in detail in this post, in brief, the circuit is:

Take care when making any connections to power lines, especially the 5 volt line on the FTDI module; if mis-connected, high currents can flow, resulting in significant damage.

The demonstration system has a Kingbright SC56-11LGWA common-cathode 7-segment display; the anodes are driven directly from the CPU I/O pins with 220 ohm series resistors. The pin mapping (reading from top left of the display) is:

Segment   CPU Pin (via 220R)
g         PB11
f         PB10
a         PB1
b         PB0
e         PB12
d         PB13
c         PB14
DP        PB15

Button    PB3 (button shorts this pin to ground)

You may wonder why I have used such a complex mapping; why not use 8 consecutive pins? The answer is that the above arrangement makes the wiring easier, at the expense of a little software complexity. This trade-off is quite common in commercial projects, where the demands of cost-saving often lead to significant complexity in the hardware configuration.

Software

The full source code is available on Github, and is compatible with Python 2.7 or 3.x. It has been tested on Windows, and is theoretically Linux-compatible, except for a problem reading data back from USB, as described in the earlier blog – this issue needs to be resolved. The Python library dependencies are:

• PyQt v4 or v5 (GPL version)
• ftd2xx
• pypiwin32 (if running Python 3.x)

These can be installed with ‘pip’ or ‘pip3’ as usual.

PyQt

The following text describes how I created the graphical front-end, for the benefit of anyone wishing to understand or modify the code; this isn’t necessarily the best way, it is just what I did based on past experience.

There are many options for creating a GUI in Python; I’ve used PyQt in the past, so that is the option I’ve chosen here. In case you’re unfamiliar with it, the current version is 5, but many installations are still version 4, so I’ve written the code to be compatible with both, even though this does involve some manipulation of the libraries:

try:
    from PyQt5.QtGui import QBrush, QPen, QColor, QFont, QTextCursor, QFontMetrics, QPainter
    from PyQt5.QtWidgets import QApplication, QGraphicsScene, QGraphicsView, QGraphicsSimpleTextItem
    from PyQt5 import QtCore, QtWidgets
except:
    from PyQt4.QtGui import QBrush, QPen, QColor, QFont, QTextCursor, QFontMetrics, QPainter
    from PyQt4.QtGui import QApplication, QGraphicsScene, QGraphicsView, QGraphicsSimpleTextItem
    from PyQt4 import QtCore, QtGui as QtWidgets

It is possible to create the entire front-end graphically using Qt designer, then just import the GUI file and it will be displayed exactly as designed, so in theory you only need to write the event-handlers for the various actions. Personally, I find this approach a bit tricky when implementing more complex GUIs, so tend to use the PyQt function calls to build the graphics from scratch.

Main Window

The main window is as simple as possible, containing only one central widget:

class MyWindow(QtWidgets.QMainWindow, MyWidget):
    graph_updater = QtCore.pyqtSignal(str)

    def __init__(self, parent=None):
        QtWidgets.QMainWindow.__init__(self, parent)
        self.widget = MyWidget(self)
        self.setCentralWidget(self.widget)
        self.setWindowTitle(VERSION)
        self.resize(*WINDOW_SIZE)
        self.graph_updater.connect(self.widget.update_graph)

A ‘graph_updater’ signal is defined so that any thread can send a request to update the graphics; this will be queued until the main GUI thread is back in control, so there is no ambiguity over which thread is performing the updates.

The single widget contains all the graphical elements, and the hierarchy is important if  they are to be displayed correctly; for example, if the window is resized, I want all the elements to be resized in proportion, and that is only possible if the correct parent-child relationship is maintained.

The upper area of the widget is graphics, the lower is text; they are combined using a vertical layout widget.

self.text = QtWidgets.QTextEdit()
self.scene = QGraphicsScene()
self.view = MyView(self.scene)
...
layout = QtWidgets.QVBoxLayout()
layout.addWidget(self.view, 30)
layout.addWidget(self.text, 10)
self.setLayout(layout)

Text display

The text area is used to display all kinds of diagnostic information, so it is convenient to redirect the console print function to display here. This is done by creating ‘write’ and ‘flush’ functions in the widget, which emit a signal linked to an updater function:

text_updater = QtCore.pyqtSignal(str)
...
self.text_updater.connect(self.update_text)
...
# Handler to update text display
def update_text(self, text):
    disp = self.text.textCursor()           # Move cursor to end
    disp.movePosition(QTextCursor.End)
    text = str(text).replace("\r", "")      # Eliminate CR
    while text:
        head,sep,text = text.partition("\n")# New line on LF
        disp.insertText(head)
        if sep:
            disp.insertBlock()
    self.text.ensureCursorVisible()    # Scroll if necessary

# Handle sys.stdout.write: update text display
def write(self, text):
    self.text_updater.emit(str(text))
def flush(self):
    pass

Now all that is necessary is to redirect console output to the main widget..

sys.stdout = self

..and magically anything in a ‘print’ function will appear in the text display. The PyQt Signal interface ensures there are no threading problems, you can still use the print function anywhere.

Graphical display

When displaying graphics, Qt (and hence PyQt) makes a distinction between the graphical objects (the ‘scene’) and their realisation on the screen (the ‘view’).

When first experimenting with background patterns, I discovered one (Dense1Pattern) that creates rows of holes similar to the prototyping board. Having fixed on this, it was logical to draw all objects with respect to this grid, i.e. in nominal units of 0.1 inches as on the prototyping board, though the graphics will expand or shrink when the window size is changed.

Drawing

To draw an object, it is just added to the scene, and it will automatically be displayed, for example to draw a circle:

# Add circle to grid, given centre
    def draw_circle(self, gpos, size, pen, brush=PIN_ON_BRUSH):
        size *= GRID_PITCH
        x,y = self.grid_pos(gpos)
        p = self.scene.addEllipse(0, 0, size, size, pen, brush)
        p.setPos(x-size/2.0, y-size/2.0)
        return p

You can see the conversions from screen-units to grid-units. One less-obvious aspect of this code is that the ellipse is originally drawn at the 0,0 origin, then moved into place, rather than being drawn at the final location; this simplifies any subsequent operations such as movement or rotation.

Animation

The drawing function returns the object that has been created, which must be saved somewhere, so it can be animated.  The most obvious form of animation is to replace that object with another, e.g. one drawn in dark red to light red, but there is an easier way to modify any drawn object: change the opacity – the extent to which the object is transparent or opaque. If a bright red object is drawn with an opacity value of 0.1, it becomes a faint dark red; changing the opacity to 1.0 restores the full strong colour.

So the objects that are to be animated are stored as a list in a dictionary, indexed on the signal name (e.g. ‘PB10’); when that signal changes state, it is only necessary to walk the list, setting the opacity as required.

# Set pin (or segment) on/off state
# Format is 'name=value', e.g.  'PA10=1'
def set_pin(self, s):
    name, eq, num = s.partition('=')
    if eq and name in self.sigpins:
        val = int(num, 16)
        for p in self.sigpins[name]:
            if int(p.opacity()) != val:
                p.setOpacity(PIN_ON_OPACITY if val else PIN_OFF_OPACITY)

7-segment display

As it happens, PyQt has a widget to draw 7-segment displays, but in this case it is easier to animate if drawn from scratch. The standard segment notation is:

To draw this in one continuous operation, I start with segment F, then ABCDEG. Once drawn, the list is rearranged into ABCDEFGH order, where H is the decimal point.

SWD interface

The SWD interface uses an FTDI USB device as described in detail here. The software used in this project is very similar, but has been optimised to scan the I/O ports quite fast, around 100 times per second. The way this has been achieved is to group all the commands to the FTDI device together, so a single block of data requests is sent out over the USB bus, then a single block of responses is read back.

All the outgoing requests are buffered, rather than being sent individually – this is quite a simple change, you just have to remember to flush the buffer before reading back the results. However you then have the problem that the returned data block consists of the data from several requests – how do you work out which data value corresponds to which request? The method I’ve adopted is to create a polling list, with objects representing the memory addresses to be polled

poll_vars = []  # List of variables to be polled

# Storage class for variable to be polled
class Pollvar(object):
    def __init__(self, name, addr):
        self.name, self.addr = name, addr
        self.value = None

# Add variable to the polling list
def poll_add_var(name, addr):
    poll_vars.append(Pollvar(name, addr))

The data requests are generated by walking down the list, then when the responses arrive, the ‘value’ fields are filled in sequentially, or set to ‘none’ if there was an error.

# Send out poll requests
def poll_send_requests(h):
    for pv in poll_vars:
        swd.swd_wr(h, swd.SWD_AP, APORT_TAR, pv.addr, True, False)
        swd.swd_idle_bytes(h, 2)
        swd.swd_rd(h, swd.SWD_AP, APORT_DRW, True, False)
        swd.swd_rd(h, swd.SWD_AP, APORT_DRW, True, False)

# Get poll responses
def poll_get_responses(h):
    for pv in poll_vars:
        swd.swd_wr(h, swd.SWD_AP, APORT_TAR, pv.addr, False, True)
        swd.swd_rd(h, swd.SWD_AP, APORT_DRW, False, True)
        req = swd.swd_rd(h, swd.SWD_AP, APORT_DRW, False, True)
        pv.value = req.data.value if (req.data is not None and
                    req.ack.value==swd.SWD_ACK_OK) else None

It is slightly confusing that the request & response use the same swd_wr and swd_rd functions; the key to understanding this code is to look at the boolean values. ‘True, False’ means that a transmission is sent, but no data is read back; ‘False, True’ is the opposite, in that nothing is sent, the response is just read back.

If you want to see the SWD transactions, try setting swd.VERBOSE to True:

SWD interface: FT232H device in Single RS232-HS
Reporta

Rd 0 IDCODE  1BA01477 Ack 1
Wr 0 ABORT   0000001E Ack 1
Wr 4 CTRL    50000000 Ack 1
Rd 4 STATUS  F0000040 Ack 1
DP ident: 1BA01477
Wr 8 SELECT  000000F0 Ack 1
Rd C DRW/BD3 00003BDB Ack 1
Rd C DRW/BD3 14770011 Ack 1
AP ident: 14770011
Wr 8 SELECT  00000000 Ack 1
Wr 0 CSW/BD0 22000002 Ack 1
Wr 4 TAR/BD1 40010C08
Rd C DRW/BD3
Rd C DRW/BD3
Wr 4 TAR/BD1 40010C08 Ack 1
Rd C DRW/BD3 14770011 Ack 1
Rd C DRW/BD3 000043D9 Ack 1
..and so on..

Looking at the last block of 6 transactions, the first 3 are a write-cycle to the SWD TAR (transfer address) register, then a dummy read and the actaul read cycle. These are the requests being sent to the target system; nothing is being read back so the read-data & ack-status values are unknown. The second block of 3 are the readback of these requests, so the actual values are displayed.

An ‘ack’ value of anything apart from 1 is an error; a value of 0 or 7 suggests the target isn’t responding, 2 means it is trying to send a delayed response, and 4 indicates a hard error – this is sticky, so will persist until cleared by writing to the ‘abort’ register.

Code structure

The ‘reporta’ project source files on Github are:

reporta.py:   main program
rp_pyqt.py:   PyQt interface
rp_arm.py:    ARM processor definitions
rp_swd.py:    SWD interface
rp_ftd2xx.py: FTDI device driver

The lower 3 files are very similar to their counterparts in these posts, basically I’ve copied the files with minor modifications. Each file has a ‘main’ function to demonstrate its functionality; for example, you can see the PyQt interface running without the FTDI hardware, just by executing rp_pyqt.py

The main ‘reporta’ program has no command-line options, currently the I/O port of interest is hard-coded; to add other ports (or CPU memory locations) to the polling list, just use the poll_add_var function.

As with the previous posts, I must state that so far I’ve only tested this technique on the STM32F103 processors; others may require custom powerup sequences, or special incantations to gain access to the CPU memory – but hopefully my Python code will provide a useful starting-point.

Copyright (c) Jeremy P Bentham 2018. Please credit this blog if you are using the information or software in it.

In part 4 we got as far as reading in the CPU identification, which is of no real use; in this part we’ll actually read some of the CPU internals, but first we need to understand how SWD accesses are controlled.

DAP: DP and AP

You may think that we just need to feed a 32-bit address into the SWD port, and get a value back, but the reality is much more complicated. The SWD clock & data lines are connected to the CPU Debug Port (SW-DP), which has its own address space. Read/write accesses to the CPU memory aren’t controlled by the DP; you have to go through the Access Port (AP), which also has a separate memory space, and is bank-switched.

DP

We’ve already accessed the Debug Port ID code register, the other read-write registers are:

# Debug Port (SWD-DP) registers
# See ARM DDI 0314H "Coresight Components Technical Reference Manual"
DPORT_IDCODE        = 0x0   # ID Code / abort
DPORT_ABORT         = 0x0
DPORT_CTRL          = 0x4   # Control / status
DPORT_STATUS        = 0x4
DPORT_SELECT        = 0x8   # Select
DPORT_RDBUFF        = 0xc   # Read buffer

These registers control various aspects of the debug interface, for example the ‘abort’ register is used to reset the internal logic after an error, and the ‘control’ register is used to power up the peripherals needed for debugging.

The ‘select’ register has bitfields to control AP and DP bank-switching; it is convenient to define this using CTYPES:

from ctypes import Structure, Union, c_uint
# AHB-AP Select Register
class DP_SELECT_REG(Structure):
    _fields_ = [("DPBANKSEL",   c_uint, 4),
                ("APBANKSEL",   c_uint, 4),
                ("Reserved",    c_uint, 16),
                ("APSEL",       c_uint, 8)]
class DP_SELECT(Union):
    _fields_ = [("reg",   DP_SELECT_REG),
                ("value", c_uint)]
dp_select = DP_SELECT()

The union allows us to specify the whole 32–bit register when doing SPI read & write cycles, but still access individual bitfields within it.

AP

The AP registers are:

# Access Port (AHB-AP) registers, high nybble is bank number
# See ARM DDI 0337E: Cortex M3 Technical Reference Manual page 11-38
APORT_CSW           = 0x0   # Control status word
APORT_TAR           = 0x4   # Transfer address
APORT_DRW           = 0xc   # Data read/write
APORT_BANK0         = 0x10  # Banked registers
APORT_BANK1         = 0x14
APORT_BANK2         = 0x18
APORT_BANK3         = 0x1c
APORT_DEBUG_ROM_ADDR= 0xf8   # Address of debug ROM
APORT_IDENT         = 0xfc   # AP identification

The Control Status Word register controls various aspects of the CPU memory accesses, for example the data size, and auto-increment for reading blocks of memory:

# AHB-AP Control Status Word Register
class AP_CSW_REG(Structure):
    _fields_ = [("Size", c_uint, 3),
                ("Res1", c_uint, 1),
                ("AddrInc", c_uint, 2),
                ("DbgStatus", c_uint, 1),
                ("TransInProg", c_uint, 1),
                ("MODE", c_uint, 4),
                ("Res2", c_uint, 13),
                ("HProt1", c_uint, 1),
                ("Res3", c_uint, 3),
                ("MasterType", c_uint, 1),
                ("Res4", c_uint, 2)]
class AP_CSW(Union):
    _fields_ = [("reg", AP_CSW_REG),
                ("value", c_uint)]
ap_csw = AP_CSW()

Because the AP can be accessing main CPU memory, it has two time-dependencies:

• the AP may return a ‘wait’ indication in the status field so the transaction has time to go though
• the data isn’t returned immediately; first you have to do a dummy read cycle, then a second read cycle that actually returns the data

So to set up a CPU memory read cycle, we need to configure the AP (including its bank-switching) then set a transfer address

# Configure AP memory accesses
def ap_config(d, inc, size):
    dp_select.reg.APBANKSEL = 0     # Zero bank
    swd.swd_wr(d, swd.SWD_DP, DPORT_SELECT, dp_select.value)
    ap_csw.reg.MasterType = 1
    ap_csw.reg.HProt1 = 1           # Enable incrementing, set access size
    ap_csw.reg.AddrInc = 1 if inc else 0
    ap_csw.reg.Size = 0 if size==8 else 1 if size==16 else 2
    return swd.swd_wr(d, swd.SWD_AP, APORT_CSW, ap_csw.value)

# Set AP memory address
def ap_addr(d, addr):
    swd.swd_wr(d, swd.SWD_AP, APORT_TAR, addr)
    swd.swd_idle_bytes(d, 2)            # Idle to avoid 'wait' response

# Do an immediate read of a 32-bit CPU memory location
def cpu_mem_read32(d, addr):
    ap_addr(d, addr)                              # Address to read
    swd.swd_rd(d, swd.SWD_AP, APORT_DRW)          # Dummy read cycle
    r = swd.swd_rd(d, swd.SWD_AP, APORT_DRW)      # Read data
    return r.data.value if r.ack.value==swd.SWD_ACK_OK else None

There are 2 idle (null) bytes after the target memory address is set. These give the AP time to process the address value before it is used; if omitted, the AP gives an ack value of 2 (‘wait’) on the next transaction.

User interface

The console-based user interface is minimal; it just allows you to specify the STM32F103  GPIO port or memory address to be accessed, e.g.

python ftdi_py_part5.py gpiob
DP ident: ack 1, value 1BA01477h
Powerup:  ack 1, value F0000040h
AP ident: ack 1, value 14770011h
40010C00: GPIOB CRL=44488433 CRH=33333344 IDR=00007FDA ODR=00007C1A

python ftdi_py_part5.py 0
DP ident: ack 1, value 1BA01477h
Powerup:  ack 1, value F0000040h
AP ident: ack 1, value 14770011h
00000000: 20005000

If a gpio port is specified, four of its register values are printed out (CRL, CRH, IDR, ODR); the other diagnostic values can be useful if the access fails. Further help is available by setting the VERBOSE flag at the top of the part 4 file, which enables a printout of all the SWD cycles:

# Command line with invalid address
python ftdi_py_part5.py 800000 
  Rd 0 IDCODE  1BA01477 Ack 1
DP ident: ack 1, value 1BA01477h
  Wr 0 ABORT   0000001E Ack 1
  Wr 4 CTRL    50000000 Ack 1
  Rd 4 STATUS  F0000000 Ack 1
Powerup:  ack 1, value F0000000h
  Wr 8 SELECT  000000F0 Ack 1
  Rd C DRW/BD3 00000000 Ack 1
  Rd C DRW/BD3 14770011 Ack 1
AP ident: ack 1, value 14770011h
  Wr 8 SELECT  00000000 Ack 1
  Wr 0 CSW/BD0 22000012 Ack 1
  Wr 4 TAR/BD1 00800000 Ack 1
  Rd C DRW/BD3 14770011 Ack 1
  Rd C DRW/BD3 00000000 Ack 4
00800000: ?

Source code

Here is the final batch of source-code. I’ve only tested the it with STM32F1 CPUs, so if you are trying to communicate with something else, there is a strong possibility you’ll need to make some changes. Points to bear in mind:

1. Check the the CPU supports SWD, and the connections it uses.
2. Check for any special power-up requirements, e.g. sending the reset sequence multiple times, or setting registers to enable debugging mode
3. Check that the SWD clock and signal lines are toggling OK.
4. Watch out for acknowledgement values 2 and 4, indicating a problem.
5. Once an error occurs, it will persist over successive cycles until reset by writing to the ABORT register.
6. Good luck!

# Python FTDI SWD CPU memory read from iosoft.blog
# Compatible with Python 2.7 or 3.x
#
# v0.01 JPB 8/12/18

import sys, ftdi_py_part3 as ft, ftdi_py_part4 as swd
from ctypes import Structure, Union, c_uint

# STM32F1 address values for testing
# Address of GPIO Ports A - E on STM32F1
ports = {"GPIOA":0x40010800, "GPIOB":0x40010C00, "GPIOC":0x40011000,
         "GPIOD":0x40011400, "GPIOE":0x40011800}
# GPIO registers at offsets 0, 4, 8, 12
gpio_regs = ("CRL", "CRH", "IDR", "ODR")

# Debug Port (SWD-DP) registers
# See ARM DDI 0314H "Coresight Components Technical Reference Manual"
DPORT_IDCODE        = 0x0   # ID Code / abort
DPORT_ABORT         = 0x0
DPORT_CTRL          = 0x4   # Control / status
DPORT_STATUS        = 0x4
DPORT_SELECT        = 0x8   # Select
DPORT_RDBUFF        = 0xc   # Read buffer

# Access Port (AHB-AP) registers, high nybble is bank number
# See ARM DDI 0337E: Cortex M3 Technical Reference Manual page 11-38
APORT_CSW           = 0x0   # Control status word
APORT_TAR           = 0x4   # Transfer address
APORT_DRW           = 0xc   # Data read/write
APORT_BANK0         = 0x10  # Banked registers
APORT_BANK1         = 0x14
APORT_BANK2         = 0x18
APORT_BANK3         = 0x1c
APORT_DEBUG_ROM_ADDR= 0xf8   # Address of debug ROM
APORT_IDENT         = 0xfc   # AP identification

# DP Select Register
class DP_SELECT_REG(Structure):
    _fields_ = [("DPBANKSEL",   c_uint, 4),
                ("APBANKSEL",   c_uint, 4),
                ("Reserved",    c_uint, 16),
                ("APSEL",       c_uint, 8)]
class DP_SELECT(Union):
    _fields_ = [("reg",   DP_SELECT_REG),
                ("value", c_uint)]
dp_select = DP_SELECT()

# AHB-AP Control Status Word Register
class AP_CSW_REG(Structure):
    _fields_ = [("Size",        c_uint, 3),
                ("Res1",        c_uint, 1),
                ("AddrInc",     c_uint, 2),
                ("DbgStatus",   c_uint, 1),
                ("TransInProg", c_uint, 1),
                ("MODE",        c_uint, 4),
                ("Res2",        c_uint, 13),
                ("HProt1",      c_uint, 1),
                ("Res3",        c_uint, 3),
                ("MasterType",  c_uint, 1),
                ("Res4",        c_uint, 2)]
class AP_CSW(Union):
    _fields_ = [("reg",   AP_CSW_REG),
                ("value", c_uint)]
ap_csw = AP_CSW()

# Select AP bank, do read cycle
def ap_banked_read(d, addr):
    dp_select.reg.APBANKSEL = addr >> 4;
    swd.swd_wr(d, swd.SWD_DP, DPORT_SELECT, dp_select.value)
    swd.swd_rd(d, swd.SWD_AP, addr&0xf)
    return swd.swd_rd(d, swd.SWD_AP, addr&0xf)

# Configure AP memory accesses
def ap_config(d, inc, size):
    dp_select.reg.APBANKSEL = 0     # Zero bank
    swd.swd_wr(d, swd.SWD_DP, DPORT_SELECT, dp_select.value)
    ap_csw.reg.MasterType = 1
    ap_csw.reg.HProt1 = 1           # Enable incrementing, set access size
    ap_csw.reg.AddrInc = 1 if inc else 0
    ap_csw.reg.Size = 0 if size==8 else 1 if size==16 else 2
    return swd.swd_wr(d, swd.SWD_AP, APORT_CSW, ap_csw.value)

# Set AP memory address
def ap_addr(d, addr):
    swd.swd_wr(d, swd.SWD_AP, APORT_TAR, addr)
    swd.swd_idle_bytes(d, 2)            # Idle to avoid 'wait' response

# Do an immediate read of a 32-bit CPU memory location
def cpu_mem_read32(d, addr):
    ap_addr(d, addr)                              # Address to read
    swd.swd_rd(d, swd.SWD_AP, APORT_DRW)          # Dummy read cycle
    r = swd.swd_rd(d, swd.SWD_AP, APORT_DRW)      # Read data
    return r.data.value if r.ack.value==swd.SWD_ACK_OK else None

if __name__ == "__main__":
    mem = sys.argv[1].upper() if len(sys.argv) > 1 else None
    dev = ft.ft_open()
    if not dev:
        print("Can't open FTDI device")
        sys.exit(1)
    ft.set_bitmode(dev, 0, 2)           # Enable SPI
    ft.set_spi_clock(dev, 1000000)      # Set SPI clock
    ft.ft_write(dev, (0x80, 0, ft.OPS)) # Set outputs
    swd.swd_reset(dev)                  # Send SWD reset sequence
    resp = swd.swd_rd(dev, swd.SWD_DP, DPORT_IDCODE) # Request & response
    if resp is None:
        print("No response")
    else:
        print("DP ident: ack %u, value %08Xh" % (resp.ack.value, resp.data.value))
        swd.swd_wr(dev, swd.SWD_DP, DPORT_ABORT, 0x1e)    # Clear errors
        swd.swd_wr(dev, swd.SWD_DP, DPORT_CTRL,  0x5<<28) # Powerup request
        resp = swd.swd_rd(dev, swd.SWD_DP, DPORT_STATUS)  # Get status
        print("Powerup:  ack %u, value %08Xh" % (resp.ack.value, resp.data.value))
        resp = ap_banked_read(dev, APORT_IDENT)           # Get AP ident
        print("AP ident: ack %u, value %08Xh" % (resp.ack.value, resp.data.value))
        ap_config(dev, 1, 32);                            # Configure AP RAM accesses
        s = ""
        if mem in ports:
            addr = ports[mem]
            s = "%08X: %s" % (addr, mem)
            for reg in gpio_regs:
                val = cpu_mem_read32(dev, addr)
                s += " %s=%08X" % (reg, val)
                addr += 4
        else:
            try:
                addr = int(mem, 16)
            except:
                addr = None
            if addr is not None:
                s = "%08X:" % addr
                val = cpu_mem_read32(dev, addr)
                s += " %08X" % val if val is not None else " ?"
        print(s)   

    dev.close()

# EOF

Next steps

Whilst this code is an interesting framework for experimentation, it lacks some of the features and error-handling of a ‘real’ application, for example it’d be nice to draw an animated picture of the CPU, showing the SWD poll results graphically. That is the subject of another post.

Copyright (c) Jeremy P Bentham 2018. Please credit this blog if you are using the information or software in it.