_drafts

Leaflet and Image Annotation Template

Thu, 05 Mar 2026 00:00:00 +0000

Someone asked me if there was a tool to add hotspot annotations to an image for a presentation where she wanted to label some features. I could swear I knew some methods to do this, like H5P Image Hotspots, StoryMapJS gigapixel option, or some IIIF thing… but I couldn’t find something simple.

A long time ago I played around with Leaflet CRS to create something similar, but the antirubbersheeter “geocoding” app I used to get the coordinates has gone away. So I decided to quickly build a replacement!

This lead to two new template projects:

image-annotate-template - image annotation with Annotorious. Add custom sized rectangle markers and display a modal or popover on click.
leaflet-annotate-template - image and map annotation with Leaflet. Uses Leaflet CRS to display an image with markers + popups or a conventional map.

The concept is that the websites contains a page for interactively creating annotations which can then be exported as JSON, which in turn is put back into the project to publish/host annotated images embedded in content pages.

Each is designed as a template project –> you make a copy for yourself to create a fully self-contained static website project that can be easily hosted on github pages or similar. The repositories contain all the assets with no external dependencies. This means you can create a copy, do your thing, and then just leave it–and it won’t break.

Developing with Claude

I am targeted in how I use LLMs in my work so I wanted to outline how I used Claude in this case to very quickly develop these mini projects:

I started from the base of bootstrap5-template which provides a sort of boiler plate Jekyll project with all the self contained assets. I like to start with a limited environment that I understand.
Next I add in the libraries I want to use, again not using a CDN or npm, just actual copies of the files to ensure it stays self contained. I don’t let the LLM decide which libraries to use or install random stuff, I find they are pretty maximalist throwing in all sorts of extra junk.
Then I set up the placeholder files in the template, this way I control the organization and ensure it makes sense for future users of the template.
I start drafting in the placeholder files, adding the asset links and script tags.
Inside the script tags I start writing comments detailing what I want. I add variables containing the data and options that I want available. I know better than the LLM where the data is in the template. In this context, I am using Liquid to pull in data from other files or Jekyll.
In the comments I add links to the library documentation, and sometimes to specific features of the libraries I want it to use.
I create some same JSON files demonstrating the output I want, in the location in the template where they will be.
Then I open the VS Code Copilot Chat. I provide all the related files as context and ask the LLM to finish writing one specific script. With this much structure and context the results are usually pretty good right away.

I have access to GitHub Copilot via a free educator account–I have no idea about subscriptions or tokens etc. I turn off all the auto suggestions and distractions in the editor, so that I can just use it by clicking the Copilot Chat or asking for inline chat. In this case I used Claude Sonnet 4.6 in “Agent” mode (check the Copilot model comparison for reference).

bootstrap5-template

bootstrap5-template is a basic template repository to create a Bootstrap 5 site using Jekyll on GitHub Pages (or where every you want to host it). The layout is based on the Bootstrap starter template example with a navbar, search box (using lunr.js), and footer. It is intended as a quick starting point for creating new web projects.

Demo

Image Annotate Template

image-annotate-template is a simple website template for creating a presentation with annotated images. It uses Jekyll static site generation, Bootstrap 5, and Annotorious. This makes it easy to host on GitHub Pages or anywhere else.

Demo

Leaflet Annotate Template

leaflet-annotate-template is a simple website template for creating a presentation with annotated images and maps. It uses Jekyll static site generation, Bootstrap 5, and Leaflet.js. This makes it easy to host on GitHub Pages or anywhere else.

Demo

Workshop Templates

Mon, 05 Jan 2026 00:00:00 +0000

My most re-used projects are a series of three workshop templates which have been adopted by hundreds of people to make workshop outlines, conference websites, lab websites, documentation, and other projects.

The idea is to provide a template repository with a basic Jekyll project set up with everything you need to start a simple website, designed for easy hosting on GitHub Pages (or anywhere).

Each repository is a template project –> to get started, make a copy and replace the demo with your own content and customizations. The content pages serve as documentation, models to follow, and examples to copy from. The user just needs to understand a bit about GitHub and how to edit Markdown–which you can learn while doing it!

Here are the three projects, in order of development:

workshop-template (2017) - the original, built on the old Jekyll Minima theme, provides conventions to create nav and footer so that you can step through the workshop content.
workshop-template-b (2018) - features a basic custom Bootstrap theme, includes to simplify adding Bootstrap components, and conventions to add features using front matter on each page (such as video, summaries, and topics).
Learn-Static lesson-template (2021) - features a sidebar navigation with nested sections, providing clear structure for step by step content. It has an advanced search for quickly finding content. It is great for documentation.

I regularly use workshop-template-b and lesson-template for my teaching and documentation projects, so they are often updated with new versions of Bootstrap and other features. The assets are all self contained and complete, so once you make a copy, it requires no ongoing maintenance or dependencies.

All content is written using basic Markdown, making it simple to write, edit, and reuse lesson materials. The templates provides Liquid includes to simplify adding features to your pages.

Writing content in this simple, reuseable format makes for a better Open Educational Resource since anyone can make a copy and adapt–importantly, I find this means myself, I can easily re-use and adapt all the content I write for new purposes. I find this much easier than making slides!

Character Corruption Issues in CSVs

Sat, 30 Aug 2025 00:00:00 +0000

CSVs are plain text files, which means all the bytes in the file are a series of character codes, each representing some text character (a, 1, <, etc) or control characters (like tabs and line breaks). The characters in a plain text file are encoded at the byte level following various standards, such as UTF-8 (which is the standard encoding on the web). The computer reads the bytes then uses the encoding standard to select which visual character to display.

Unfortunately, character encoding is complicated!

For example, a text file containing one encoding can get a character from a different standard saved into it. A text file written in one encoding can be opened with the wrong encoding. These cases can end up with the characters in the file being incorrectly displayed or fully corrupted (saved incorrectly into the file).

When that happens, usually what is displayed is the replacement characters:

���

�

Or nonsense like:

Singaporeâ€™s

(This nonsense is called Mojibake. A slightly different issue is when a specific font does not have the character to display, which is called “tofu”–an issue with the font, not necessarily encoding.)

Avoiding character corruption issues requires careful handling and attention to CSVs.

The main forms of corrupt characters are from two sources:

Opening UTF-8 CSV in Excel (or opening it in wrong character encoding in another program)
- This situation can be avoided by NEVER opening or saving a CSV with Excel! Only open and save XLSX files with Excel.
- If you accidentally double click on a CSV and it opens in Excel, the characters are potentially corrupted (via the replacement character issue).
Ambiguous characters saved in UTF-8 later opened by a spreadsheet program (ambiguous characters converted into replacement characters)
- This situation usually comes from pasting text from MS Word or other document editor into a textfile, CSV, or Sheet. At first it will appear correct, but when re-imported into Sheets all ambiguous character will be corrupted into replacement characters.

Ambiguous characters

Ambiguous characters are generally special fancy looking punctuation marks that are supported by specific editing programs, but not necessarily UTF-8 itself. The main examples are smart quotes, smart apostrophes, emdashes, and ellipsis:

”
“
–
—
’ 
…

For example, when writing in MS Word or other editors, typing a pair of quotes " are automatically replaced with smart quotes, “example”. These have specific meaning in Word, but not in UTF-8.

The source of these characters in our metadata is most likely from pasting text from another program, such as MS Word.

These ambiguous characters normally display fine in VS Code (they are usually highlighted with a gold box indicating an ambiguous character), and can be built into the website without any issues. They can go through OpenRefine without issues.

However, when importing into Google Sheets, they will become corrupted (i.e. the ambiguous character byte sequence will be replaced with the replacement character code in UTF-8). Once corrupted they will display as tofu ��, and since the byte code has been replaced, it is impossible to translate back to the original.

It is often very difficult to notice this corruption as it might only impact a few random values in a large spreadsheet.

If a CSV has ambiguous characters, it is best to try to clean them up. Work in VS Code or OpenRefine, do not import into Sheets to do this work!

Use the regular expression [^\x00-\x7f] to find problem characters.
Use find & replace to replace the ambiguous character with a correct UTF-8 version
In OpenRefine, use All column > Edit all columns > Replace smart quotes with ASCII…

Special characters

Metadata often contains accented characters or diacritics, such as:

à, á, â, ä, å, æ, ç, è, é, ê, ë, ì, í, î, ï, ñ, ò, ó, ô, ö, ø, ù, ú, û, ü, ý

These can commonly become corrupted by incorrectly handling UTF-8 or pasting from non-UTF-8 sources. Diacritics are the most commonly corrupted character type when opening a CSV in Excel.

Pay close attention to diacritics in metadata to ensure they do not become corrupted–once corrupted it is difficult to recover the correct content.

Numbers converted to decimals

Another form of corruption can happen when importing CSVs into spreadsheet programs if the option to “Convert text to numbers, dates, and formulas” is NOT UNCHECKED. In columns that have numbers, sometimes spreadsheet program will interpret the column as a number rather than text. In that case it can add a decimal (e.g. 1 to 1.0) or round off as if it was a decimal (e.g. id 1.10 to 1.1).

This often comes up in identifier columns that use chapter or folder numbers like 1.1, 1.2. You will end up with several 1.1 values as 1.1, 1.10, 1.100 will all be converted to the same number.

Escaped unicode

In some applications, uncommon characters (anything unicode outside of basic ASCII) can be replaced with “escaped unicode”. This is more common in JSON. To handle correctly, the software will need to unescape the unicode converting it into a normal UTF-8 character (which is not typically done with spreadsheet CSVs).

For example, Montréal Canada:

"value": "Montr\u00e9al Canada"

In this case the UTF-8 character é has been replaced by the unicode \u00e9 (with the escape \ character in front).

Escaped unicode can be corrupt as well. For example:

`Singapore\u00e2\u0080\u0099s`

In this case, the original unicode for apostrophe has been corrupted into a nonsense three character sequence.

ScanTailor Book Scan Image Processing

Fri, 29 Aug 2025 00:00:00 +0000

ScanTailor is a unique piece of software that does its niche job very well–so I wanted to gather some notes about it.

There was a ton of enthusiasm in 2000 - 2010s for DIY book scanning, where you set up your own library lab style book scanning rig and join Internet Archive in digitizing the worlds print heritage. A lot of this is embodied in the DIY Bookscanner community which contains a lot of information, plans, and documentation.

Once you capture images of a book’s pages (either with a rig or flatbed scanner), you need to process them into something useable. There is two competing aims of the post-processing:

Create super clean images for OCR to extract the text for a fully digital edition (the images won’t be used for human reading)
Normalize and clean images for creating a PDF facsimile edition (make something that looks like the original book pages, which limits how much you will process the images. Images become the PDF pages.)

Some commercial scanning rigs have their own proprietary software, and others use Acrobat, Abby Finereader, or Omnipage (which aren’t really designed for this task). The DIY community created a few pieces of software specially suited to the post-processing challenges–resulting in the original ScanTailor (probably don’t install this version!). But in the last decade, community energy has fizzled (even at its height was very tiny and niche)–leaving a everything unmaintained.

Since then ScanTailor has been forked and re-forked dozens of times by a bunch of random people trying to enhance and keep it functioning, leaving a maze of versions that is hard to navigate.

The current most solid option seems to be ScanTailor-Advanced. A couple that seem active, but I haven’t personally used are scantailor-experimental and scantailor-universal).

What ScanTailor does well is providing a mix of automated batch processing with individual review / manual tweaks on each page. Through each step of processing it gives you a preview of the results–then when you want to finalize, it runs all steps in one operation to maximize image quality.

It can:

Split, rotate, and crop pages.
Deskew.
Detect page content, so you can select text blocks and illustrations. The text and illustrations will be processed differently (which is pretty hard to do using other tools), and everything else cropped away.
For text areas, sharpens text and removes the background.
With everything selected, create a standardized page size and margin so the output images are all exactly the same.

Pasty Awareness Week

Wed, 30 Jul 2025 00:00:00 +0000

Pasty Awareness Week was one of my first personal static web projects on GitHub. I started from an old version of the default Jekyll minima theme and added some content that is near & dear to my heart.

Creating this pretty random project was a great way to learn more about Jekyll and GitHub Pages when I was first starting out. The platforms and my development has evolved a lot since then, but the content is definitely still really good and important.

If you don’t know, pasties are a delicious food commonly found in the Upper Peninsula of Michigan and the Iron Range of Minnesota (and a few other US mining areas like Butte, Montana). In the UK they think Cornish folks invented them or something, but the things in the UP are uniquely their own and much better.

Be sure to celebrate National Pasty Awareness Week every year and share #PastyAware!

Correct Math Answers

Wed, 30 Jul 2025 00:00:00 +0000

Encountering the every increasing absolute crap AI web content ecosystem and thinking about LLM lack of ability to do math randomly inspired me to make this useful and absolutely essential project:

Correct Math Answers

I hope to add static generated pages to better allow AI bots to crawl and capture all the information to help train their models, but I enjoyed getting this rapidly set up, starting from bootstrap5-template and using GitHub Copilot to quickly write everything.

Like my other random projects, this was essentially a learning opportunity to figure out how to better use Copilot efficiently. I think having the LLM inside a well-known, self-contained environment is helpful to get better results than starting from scratch or letting it add batches of random dependencies.

From the site:

“Do you need to know 2 + 2, 6 - 4, 10 * -5, or 99 / 678493? We’ve got you covered! And you don’t need a calculator. If you need correct answers to math questions this site will provide exactly what you need.”

Be sure to read the About page.

Water of the Wondrous Isles

Tue, 29 Jul 2025 00:00:00 +0000

I’ve always enjoyed the writing of William Morris, a true “renaissance man” who dabbled well in an amazing array of artistic, academic, and social pursuits. Building on my Aladore projects, I was interested in creating some projects with Morris texts to further explore creating useable web editions with added data features using static web generation.

In 2020 I created morris-wondrous-isles-book project building an edition of The Water of the Wondrous Isles (1897). The intent was to create a fully self-contained, minimal static web-based “artisanal” digital edition. With the text parsed into semi-structured markdown files, there is new opportunities to generate data and search.

Water of the Wondrous Isles is a fascinating book, a female protagonist, powerful women, strange magic, in a fairy tale like setting. Give it a read at morris-wondrous-isles-book!

Beowulf-s

Tue, 29 Jul 2025 00:00:00 +0000

In 2021, I created the Beowulf-s project essentially by using OpenRefine to parse the text of Chauncey B. Tinker’s The Translations of Beowulf: A Critical Bibliography (1903) into structured data. Tinker as data provides a handy curated list of (now) public domain texts, so I enriched the data with links to digitized copies of each record on the web. I could then use the structured data (a CSV) to generate a website (using CollectionBuilder) to explore the historic bibliography of Beowulf translations.

The whole project was inspired by Maria Dahvana Headley’s introduction to Beowulf: a New Translation (2020), which I would recommend giving a read! I describe it all a bit more on the About page.

Digital Aladore and aladore-book

Tue, 29 Jul 2025 00:00:00 +0000

I started the Digital Aladore project in 2014 as a grad student project blog with the idea to take an obscure digitized public domain text and make it more usable. In 2018, I used the text to develop the aladore-book project, exploring ideas of using static web generation to create ebooks with added data features.

Here is an introduction adapted from my presentation “Digital Aladore: Reflecting on Five Years of a DH Learning Project” at DHSI Colloquium 2020:

Abstract

Digital Aladore started in 2014 as a project blog with the idea to take an obscure digitized public domain text and make it more usable. Grappling with the crummy digital legacy of the Million Book Project/Universal Library digitization initiatives of 2000’s, it was conceptually an opportunity to apply a background in Classical Studies to investigating the twisted details of digital witnesses and textual transmission. However, encountering the limitations of digitization that failed to take advantage of the affordances of digital mediums, it led down a path to learning the software and approaches for transforming images, text, and data. Importantly, publicly articulating the learning process of encountering challenges and discovering solutions resulted in a lasting resource of accessible explanations about using technical tools–attracting significant use from outside of academic sources. After more than five years of maintaining Digital Aladore as a venue for thinking through ideas and tinkering with DH tools, it has moved far beyond its origins as a course assignment in library school. The experience guided my development into the realm of open software, content, and communication, while providing a playground of data for tooling up DH projects. This talk reflects on the unexpected value, outcomes, and sustainability of a simple side project for learning DH hands-on.

Presentation

Rather than introducing my official scholarship–this presentation reflects on more than five years of a Digital Humanities side project–a little personal gem that has sat simmering on the side of my desk since grad school–and the unexpected value that comes of having this space for thinking through ideas and tinkering with DH tools. This little project acts as a medium for exploration and learning, a theme DH thinkers have commonly returned to, “messing about” and play, which is especially appropriate in the DHSI context where traditionally we come together for a week of exploring, learning, and concocting new ideas in lovely Victoria–which I am missing very much this year.

Digital Aladore was a project I started with the idea to take an obscure digitized public domain text and make it more useable. It has it’s roots in a weird mix of academic ideas, personal whimsy, and, of course, a class assignment.

In 2014 I was taking a course at UBC nicknamed “OK MOOC”. It was a collaboration of six international universities offering in person, for credit classes, embedded inside a free online course with over 12,000 other learners– it was basically very intense, overwhelming, but truly fascinating.

Digital Aladore came out of that mix, thinking about open culture, working with public domain text and open source software–as well as an aim for transparency and clear communication.

My background was in Classical Studies, meanwhile I was working in a digitization lab, I was fascinated with how the scholarship of textual transmission might apply in the digital archives. Particularly, in the legacy of terrible metadata and horrific OCR from mass digitization projects of the late 2000’s, such as the Million Book Project/Universal Library.

You could find these digitized texts online, but due to the poor quality they are barely useable for reading or analysis. On the other hand there are academic projects such as the William Morris archive that provide critical edition quality text, but are often locked in outdated web sites (the Adobe Flash Edition). I mention William Morris here because Aladore is basically an imitation his romances–but Morris is popular enough to have good human-edited reading editions available in places such as Project Gutenberg. For more obscure texts, we are stuck with trying to read auto-generated bad OCR ebooks or a lousy PDF.

I really just wanted to read a nice clean copy of Aladore, so I set out to make my own, learning everything that went into it along the way.

In retrospect, this is a classic digital librarian idea–untangle metadata, create more useable access, and reporting out about everything.

So the blog was just that–step-by-step notes about open source tools and processes to work with digitized texts, publicly articulating my learning process and recording solutions.

I processed the two print editions of Aladore that were available, and then I collated them to help in the textual editing process.

I wanted to engage people with thought about textual transmission and concepts of openness. However my most popular posts ended up being the practical steps to use OCR. And they seem to get used–for example, I ran across this forum post of an IT person trying to optimize some batch script based on a Digital Aladore post.

Think about that, some publishing company is using a random code copied from a DH student learning blog…

Documenting learning is powerful.

That approach of open communication has stuck with me, contributing to organizations such as Software Carpentry or Programming Historian, and my own workshop resources.

The text of Digital Aladore lives on.

For me, it acts as a well known text data set for use in testing out ideas or any new tools that comes across my plate, a playground of data for tooling up for other, more important, DH projects. And a lot of the use cases end up built into my current edition, aladore-book project, where I play off the “collections as data” concept to make a variety of handy text derivatives readily available for use on the web.

And, true to the original project, it is also reader friendly, so if you you are looking for a obscure summer read, check it out.

University Wifi on Linux

Mon, 02 Jun 2025 00:00:00 +0000

Sometimes connecting to secure wifi at institutions like universities is not so obvious and easy on Linux, it doesn’t just connect but asks you for a bunch of configuration options.

Go to your full Settings > Wi-Fi, click on the network name, which will get you to a security settings form with a lot of options. In my experience, if you are trying to connect to most university wifi or eduroam, this is usually the correct configuration:

Security: WPA & WPA2 Enterprise
Authentication: Protected EAP (PEAP)
Anonymous identity: [leave blank]
Domain: [leave blank]
CA certification: None
check off “No CA certificate is required”
PEAP version: Automatic
Inner authentication: MSCHAPv2
Username: [your individual username or email, for eduroam this is your full home institution email]
Password: [your individual password]

Since your password is saved, don’t share this with other users on your system.

Ubuntu 24.04

Mon, 20 Jan 2025 00:00:00 +0000

When I started writing this blog, I had new computers, and liked to install every new release of Ubuntu and Fedora to see how much better my laptops got. I still use those same computers which are now old. I only install long term releases (LTS, five+ years of support) and keep using it for a long time.

Well, Ubuntu 24.04 has been out for a while now, and I haven’t written an updated “getting started with Ubuntu” post for five years–so I thought I better get back to it!

This is (like other posts) notes to myself, since I installed Ubuntu 24.04 on a bunch of Dell XPS 13 laptops. The Dell laptops are well supported, since they used to sell them with Ubuntu as an option. They have only 8 GB ram and were running horribly slow on end of life Windows 10–but work great in Ubuntu 24.04!

Unencrypt Bitlocker

If you are installing over an old Windows machine which used Bitlocker encryption, first you may want to disable/unencrypt the drive. This step shouldn’t be necessary, however, the Ubuntu installer would NOT let me overwrite the encrypted Windows drive.

Open Start Menu and search for “manage bitlocker” (or go to Control Panel > System and Security > Bitlocker Drive Encryption)
Next to each drive it will have a “Turn off Bitlocker” option. Click it.
Wait. Unencrypting the drive takes a very long time.
Once complete, fully shutdown Windows.

Create Install USB

download Ubuntu 24.04 LTS
Burn ISO to a USB stick
Plug in your live USB into computer (while turned off)

Set up BIOS

This part varies a lot on different computers and isn’t always necessary…

Click the power button of you computer
Hit the boot interrupt key to enter BIOS menu. This is usually F2, F12, Delete, or Esc, sometimes the key is flashed on the screen as the computer boots along with the manufacturer logo. For Dell XPS hit F2.
Enter BIOS: Look for something about “SATA Operation” and select “AHCI” (you need to turn “RAID” mode off because it is not supported by Linux).
Save and exit BIOS

Boot USB

Reboot and hit the boot interrupt or boot option menu key. For Dell XPS hit F12. This will get you to a temporary boot menu (alternatively, these options are available in the main BIOS menu of most computers).
Choose the first USB boot option
After some thought, you may see an Ubuntu boot option menu, select “try and install” to continue. Or it will boot into the Ubuntu live image, where you can choose to test or install.

Install Ubuntu

Use the installer to follow through the steps (official Ubuntu tutorial). Suggestions:

Connect to wifi to greatly speed getting everything set up.
Use “Default selection” for apps, rather then the full install, unless you want LibreOffice all set up and ready.
Check the “recommend proprietary software” to simplify set up.
On “How do you want to install Ubuntu?” use the “Erase disk” option. Click “Advanced feature” and select “LVM and encryption” option. If you use encryption, you will set a encryption password (separate from your user password) that you will need to use each time your reboot the computer.

Get Set Up in Ubuntu

Update everything with the “Updates” app or in the terminal sudo apt update then sudo apt upgrade.
Open Settings:
- Settings > Appearance - set your background and new option to set an accent color.
- Settings > Ubuntu Desktop - Customize the Desktop icons and Dock. Option to turn off Enhanced Tiling (I’m not used to it, so sort of drives me crazy).
- Settings > System > Date & Time - use “Time Format” option to change 24hr or AM/PM.
Install some essentials to customize the desktop: sudo apt install gnome-tweaks gnome-shell-extension-prefs gnome-shell-extension-manager
- gnome-tweaks lets you customize all details of the desktop, for example moving the window buttons from right to left corner.
- gnome-shell-extension-prefs provides a visual tool to customize the options of gnome extensions.
- gnome-shell-extension-manager provides a visual tool to install gnome extensions.
Open “Extension Manager”, click Browse, search for “Caffeine”, and click install. Caffeine extension allows you to disable screen lock manually, and shows up in the menu from upper right corner of the menu bar.
Enable firewall: sudo ufw enable
Development essentials: sudo apt install build-essential git curl
Chromium (browser): sudo snap install chromium
VS Code (editor): sudo snap install code --classic
KeePassXC (password manager): sudo snap install keepassxc
Ruby: rbenv
python, venv or uv

Note: I don’t think tlp / advanced laptop power management is necessary any more.

Git Credential Manager

Mon, 30 Dec 2024 00:00:00 +0000

Git for Windows comes bundled with Git Credential Manager (gcm) which makes using git on command-line much easier, automatically opening up authentication in a GUI window or web browser and storing the results. On linux things are not quite so simple–however, you can now install Git Credential Manager manually.

(alternatively, you can use Git Credential Helper if you have authentication tokens set up already)

Install git-credential-manager

Follow the instructions for your OS documented at the gcm github repository. Here a the summary for Ubuntu:

download latest “.deb” package release
install the deb sudo dpkg -i <path-to-package>
run git-credential-manager configure
configure a credential store. I use the secretservice option which is built in to Ubuntu: git config --global credential.credentialStore secretservice
Restart your system! (otherwise you will get weird errors)

Next time you clone or pull from a private repository (or push), gcm will pop open a little window to handle authentication for the platform. The credentials will be stored for future re-use.

If you need to update your credentials, the easiest method is to use “Passwords and Keys” / Seahorse app: Look in Passwords > Login category. You should find an entry such as “git:https://github.com/” that corresponds with the credential. I usually just delete the old one, then provide the updated version on my next git operation.

Ruby rbenv on Ubuntu

Thu, 05 Dec 2024 00:00:00 +0000

For a long time I used RVM as version manager to install Ruby on Linux (note). However, while teaching workshops Mac users seem to run into endless issues trying to install RVM. For Mac, using brew to install rbenv seems a better alternative. Also, some folks don’t like how RVM handles it’s Ruby version switching magic (e.g. basically it is too heavy and hacks the cd command, see summary). So eventually I switched to rbenv on Linux as well.

Unfortunately, rbenv documentation is poor and seems focused on Mac users–thus this note to remember what I did on Ubuntu…

There are a few things to keep in mind:

If you have RVM currently installed, you will have to fully remove it which isn’t very easy (I switched to rbenv only when I did a fresh install).
Unlike RVM, rbenv only switches Ruby versions, it doesn’t install/build them. This is not obvious for Mac users since installing rbenv via brew automatically installs ruby-build at the same time (without mentioning it…). So you will want to install ruby-build as well.
Using rbenv has changed a bit since earlier versions (easier now), so looking at old suggestions / instructions is probably a wild goose chase.
You will have to keep rbenv and ruby-build up-to-date manually, which is a bit of a pain, but sort of refreshingly simple…
You can automate install using rbenv-installer, which is basically a script just like brew recipes (I did not try this).

Note: there is an Ubuntu package to install, so you can use sudo apt install rbenv ruby-build. Unfortunately the ruby-build package is super out of date, so it will only list very out-of-date versions of Ruby to install. This is not a good install method!

Install rbenv and ruby-build

First, install build dependencies. Honestly, I am not sure about these, but this is the suggested build environment from ruby-build, which I updated a bit for recent Ubuntu since some of the listed packages are out-of-date:

sudo apt install autoconf bison build-essential libssl-dev libyaml-dev libreadline-dev zlib1g-dev libncurses5-dev libffi-dev libgdbm-dev

Finish the install, close your terminal and open a new one.

Second, clone rbenv into your home directory:

git clone https://github.com/rbenv/rbenv.git ~/.rbenv

Third, run the init function to set it up:

~/.rbenv/bin/rbenv init

Close terminal and open a new one.

Fourth, clone ruby-build into the rbenv plugins directory:

git clone https://github.com/rbenv/ruby-build.git "$(rbenv root)"/plugins/ruby-build

Install a Ruby

Check the Ruby download page to find the latest stable version or your project requirements to find your desired Ruby version number. Check rbenv install -l to get a list of available versions.

Use rbenv install + version number, e.g.:

rbenv install 3.4.1

This can take awhile since ruby-build will download and build from source.

Once complete, set the version you want to use:

rbenv global 3.4.1

Now, ruby -v should report what you just set. If you need a different version for a specific project, run rbenv local <version number> in the directory (which creates a file called .ruby-version to keep track).

Note: Gems you install will be specific to the currently used version (global or local). When you install a new gem that includes commands (e.g. Jekyll) rbenv shims have to be rebuilt. This is done automatically (assuming you followed the rbenv init instructions), or can be manually triggered with command rbenv rehash.

Switching Rubies

rbenv versions lists installed versions with * next to current global version

rbenv global <version number> will switch global version. Once you switch, just remember that your installed gems are specific to a version, so you won’t have the same set of gems available!

Update rbenv and ruby-build

Since you installed the tools using git, to update rbenv and ruby-build you simply git pull the most recent master branch code.

cd ~/.rbenv && git pull
cd "$(rbenv root)"/plugins/ruby-build && git pull

Note: the list of available Ruby versions (rbenv install -l) is NOT automatically updated. So you should periodically update ruby-build following the git pull method above (i.e. cd ~/.rbenv/plugins/ruby-build && git pull).

Ubuntu 20th Anniversary

Thu, 10 Oct 2024 00:00:00 +0000

I was surprized to read that it is Ubuntu’s 20th anniversary–honestly, that’s pretty cool.

There are not a lot of new features for version 24.10 marking the milestone, but seems like improvements to the installer–which is fitting, because I think what was so great about Ubuntu in the early days was having a relatively easy to use install process, enabling people to actually get started on linux easier.

I’ve used Ubuntu since almost the beginning! I was a poor college student. I could pick up discarded old laptops on campus for free. The batteries were bad. Windows was bad and making them unusably slow and cumbersome. But I could plug in a usb drive, install Ubuntu, and have a great running computer. Something that just worked, had no sketchy bloatware, and fit my budget of zero.

People still complain that Linux is too weird and hard or tech elitist–I never quite understand. I was a Classical Studies student. I didn’t have any technical training or interest in computers. I worked construction and as a studio artist, so I guess I was willing to be hands on to figure something out.

This allowed me to actually have a functional computer. Once installed, it was just a perfectly normal computer, easy to use.

My partner has been an Ubuntu user now for 15 years, and she seriously has no idea what Linux or Ubuntu is. It’s never come up. If you said “operating system” she would have no clue what you mean. It’s just a computer that works great.

Honestly, that’s pretty amazing. So, thank you Ubuntu and happy anniversary.

If you want to learn more, Try Linux!

Chromebooks Disable Touchscreen

Sat, 05 Oct 2024 00:00:00 +0000

I have several nice higher end Chromebooks that we got via pandemic money for remote schooling. The problem is that in most of the devices (2 out of 3) the touch screen goes wacky, causing all sorts of ghost clicking that makes it totally unusable. Which is too bad, since the devices are fine otherwise…

I wish Chromebook manufacturers wouldn’t do such a terrible job on this hardware–these aren’t cheapo ones, so it is really disappointing. Also, not sure why disabling touch screen isn’t a built in short cut or why there isn’t a permanent method to do so (and that is on ChromeOS)…

But, if you have them, and the screen starts going crazy, there is still a couple of ways to use it.

Connect to Monitor/TV

First, to escape wild behavior if the touch screen is making it impossible to even log in, connect the Chromebook to a monitor or TV. Hopefully your device has some port that will allow that (mine all have HDMI output).

Leave the Chromebook closed, connect the monitor, a keyboard, and mouse. Then push the power button to start the device up without opening the screen.

This will allow you to get into the settings to fix things.

(Or just keep using it as a desktop computer…)

Use Chrome Flags

Open Chrome browser on the Chromebook.
Put chrome://flags/ in the address bar.
Search for ash-debug-shortcuts
Switch ash-debug-shortcuts option to “Enabled”
Click Restart (which will restart the device)

Now you will have a new keyboard shortcut Shift + Search + T that toggles the touch screen on and off.

Unfortunately, it will toggle back on each time you restart (for example when installing updates). This is okay if you can get the shortcut applied fast enough when logging in before the ghost action gets too wild.

However, if your touch screen is really out of wack, this can still be super frustrating. One of my devices is super erratic, sometimes fine I can toggle off the touch screen and log in and be happy–other times the ghosting starts zooming in and out and randomly clicking all over resulting in all kinds of wild activity. To recover, you can always go back to connecting to a monitor–but if this is the case you might want to go more drastic.

Developer Mode

If you want to go further, you have to switch on Developer Mode. This will erase all the local data and set up.

Unfortunately, Chromebook is a maze of proprietary weird exceptions, so pretty much nothing you find online with exactly match up with your actual device. It goes out of date fast and each vendor has a bunch of odd custom stuff.

enable developer mode
permanently disable touchscreen via shell - however, this does NOT work for me (xinput not found) and I have been unable to debug or find any other suggestion.
Install something else, chrultrabook and mrchromebox.

Some handy key shortcuts:

Open crosh shell: Ctrl + Alt + T
Switch to VT-2 shell with sudo commands: Ctrl + Alt + ->/F2 (log in as “chronos”)
Switch back to crosh: Ctrl + Alt + <-/F1

Firefox Tips

Fri, 05 Jul 2024 00:00:00 +0000

Tips about Firefox:

Settings > Privacy & Security > check “Use a primary password” and set. This ensures your saved passwords are encrypted on your device.
Settings > Privacy & Security > Website Advertising Preferences, uncheck “Allow” option!
Add Firefox Multi-Account Containers, Facebook Container
Add adblocker such as uBlock Origin

Disable Gestures

I have issues accedentally going back via the default “guestures” for “swipe” somehow causing Back at random on my touch pad. Turn off the option to avoid this issue:

in address bar, type about:config
accept the risk warning
search for browser.gesture
in the .swipe.left and .swipe.right options, click the edit icon, delete the value, and click the check icon to save.

Issues with Cisco VPN

For some weird reason, Cisco AnyConnect / Secure Client has issues with Ubuntu + Firefox for authentication. To avoid issues, first go to Ubuntu Settings > Connectivity and uncheck the option. Second, use Chromium for authentication…

Hoppy Alternative Drinks

Sat, 08 Apr 2023 00:00:00 +0000

I love IPA beers mainly just for the incredible hop aroma. But one downside (sometimes) to IPAs is that they tend to be high in alcohol. Sometimes you just want some delicious hops!

This table relates my experiences exploring non-alcoholic hoppy drinks. I purchase the beverages (no sponsors!), pour them into a pint glass (to get the max aroma), and drink them (to probe their tastiness). Drinks are rated 1 - 5 on these highly scientific qualities:

hoppyness - overall hop quality and quantity.
aroma - does it smell wonderfully hoppy?
body - does the beverage have mouth-feel and body?
beeriness - does it taste like or stand in for a beer? This isn’t necessary for deliciousness, but sometimes you want a beer-like substance.
deliciousness - just how delicious is it overall as a hoppy alternative beverage?

Hoppy Drink List

Sort by Rating:

Calm One HopTea, Hoplark

2 2 0 4

This is probably the most delicious HopTea, the sweet smoothness of chamomile suits the flavors very well. Citra hops.

Really Hoppy One HopTea, Hoplark

3 4 0 2

The flavors of this aren't very balanced, I find it overly hoppy but not delicious.

Green Tea One HopTea, Hoplark

2 4 0 3

Pretty good balance of very light beverage and good hop aroma. Mosaic hops.

Mile-High-Biscus One HopTea, Hoplark

2 4 0 3

Surprisingly delicious, I didn't expect to like the hibiscus, but it adds nicely to the aroma.

Hop Water, Sockeye

2 2 0 4

Very simple hoppy sparkling water, light aroma, good taste, refreshing. Grapefruit-y? Idaho grown hops.

Hop Splash, Sierra Nevada

2 1 0 2

A tasty sparkling water, less citrus-y or sweetness, crisp and refreshing. The aroma is light, and the favor isn't totally balanced. Citra and Amarillo hops.

Trail Pass IPA, Sierra Nevada

3 2 3 3

Very good flavor, but light aroma and light lager-like taste. This is the first NA IPA I have come across in 12 packs!

Hazy IPA, Sierra Nevada

3 2 3 3

Distinctive hazy favor. I think I prefer the regular Trail Pass, but this is quite good.

Brewveza, Sierra Nevada

2 1 3 2

Reasonable light lager-like drink.

Hoppy Refresher, Lagunitas

1 0 0 2

Very citrus-y more than hoppy, definitely refreshing and tasty as a sparkling water--but not really hop forward.

IPNA, Lagunitas

2 1 2 3

Light, crisp, has good bitterness but weak on the aroma which brings down the feeling of hoppyness--however, much more tasty than an average non-alcoholic beer.

Juicy IPA, Grüvi

3 4 3 3

Marketed as non-alcoholic craft brew, this goes for beeriness and does a good job. Interestingly, the taste and aroma in the first few minutes is GREAT, but then it seems to gradually get sweeter and less delicious as it warms.

Stout, Grüvi

1 1 4 4

Really tasty rich stout.

Non-Alcoholic IPA, Fremont

3 2 3 2

Pretty solid as a basic very light IPA substitute. Tasty, but almost more like a lager in favor.

Run Wild IPA, Athletic Brewing Co.

3 4 3 4

This is pretty delicious as an light IPA.

Free Wave Hazy IPA, Athletic Brewing Co.

3 3 3 4

Refreshing, tasty light beer-like light IPA. Crushable.

Lime, HOP WTR

1 1 0 1

Billed as sparkling hop water with adaptogens and nootropics, this is the most expensive drink I have tried. I found it not good. Tasted exactly like a cheap lime bubble water, but slightly nasty.

Classic, HOP WTR

0 0 0 0

Like the Lime version, I got this on 50% off sale (otherwise far too expensive) and found it nasty. It is very acidic and mineral-water-like, with very faint if any hop related favor. The main ingredient is ascorbic acid (120% of your vitamin C, rather than hops...), which I guess must contribute significantly. I didn't even finish drinking a single can--however, I must admit that for some reason my partner loved it--so maybe its just me?

Hop Water, Trail Magic

2 3 2 4

This is a delicious beverage, with 3mg THC (in Minnesota). It has a nice aroma and some body, not trying to be an beer exactly, but something delicious with hops in it.

Organic Session IPA, Sober Carpenter

4 4 4 4

A very tasty IPA substitute, basically just like a session IPA. Available in Canada.

0.0% Premium Pils, Bitburger

1 1 3 1

One of classic popular german beers as an NA. Pretty traditional sort of NA, meaning pretty terrible--but I guess better than really old school ones from the 2000s.

Original NA, Clausthaler

1 1 2 1

Really not a tasty beverage from my point of view. Something pretty bad about these old school NA beers.

Fresh Squeezed Non-Alcoholic IPA, Deschutes

4 4 3 5

Very hoppy, yet balanced good flavor. To me it seems not quite as beer-like as the Athletic Brewing offerings, but in a good way, as a slightly different genre of deliciously hoppy drink (that is similar, but not quite like beer).

Sparkle Hops, Pelican Brewing

2 2 0 3

Very good, more mineral-water-y taste.

Crazy Cake Recipe

Wed, 25 Jan 2023 00:00:00 +0000

My favorite cake (“minimal cake infrastructure”), regularly cooked after school when a teenager. Very delicious and very simple to make. Since this recipe always comes out well it is also great for experimenting by adding random other ingredients.

Heat oven 350 degrees.

Combine all dry ingredients directly in an ungreased 9 x 13in pan:

3 cups flour
2 cups sugar
1/2 cup baking cocoa (you can play with this amount depending on how strong you like your chocolate, approximately from 1/3 cp to 2/3 cup, or even 0)
2 teaspoons baking soda
1 teaspoon salt

After mixing the dry stuff in the pan, make three depressions and add each liquid ingredient into is own spot:

2 teaspoon vanilla extract
3/4 cup vegetable oil
2 tablespoons vinegar

Next pour:

2 cups cold water

over everything in the pan. Mixing everything up with a fork (blend it up, but don’t beat or over mix).

Put pan in the oven and bake for approximately 34 - 40 minutes until pick comes out clean.

I generally like to serve it up right away just plain with some milk. But you can treat it just like any other cake (e.g. cool and frost it, etc).

Deploy a Jekyll Site with GitHub Actions using Starter Workflow (v3)

Tue, 10 Jan 2023 00:00:00 +0000

GitHub Pages’ default build process runs an older version of Jekyll and does not support plugins (see dependency versions for details). However, you can set up an alternative build using GitHub Actions. GitHub has recently made setting up Actions to deploy your site to Pages easier, so this is a great option. (note, an older more manual method to set up Actions also works if you want customize more)

Prep Project Repository

First, check your “_config.yml” to ensure the url and baseurl values are set correctly for hosting on GitHub Pages following the pattern url: https://username.github.io and baseurl: /repository-name. If you don’t have these set, the Liquid filter absolute_url won’t work correctly.

Second, make sure your project has a “Gemfile” committed. A Gemfile with just this will work:

source 'https://rubygems.org'

gem 'jekyll'

Optional: Commit your “Gemfile.lock” to ensure the build uses the same setup as you have been using to develop the project. I usually .gitignore the “Gemfile.lock”, but if you want to keep the dependencies locked, you can commit it.

Note: some accounts may have GitHub Actions disabled by default. If you do not see the “Actions” tab in your repository’s navigation (in between “Discussions” and “Projects”), it will need to be turned on first. Visit the repository’s “Settings”, click on “Actions” in the left side nav menu, select “Allow all actions”, and click “Save”.

Setup Action YAML using Pages

To add a GitHub Action, you will be adding a YAML file describing the build steps to your project repository–you can do this automatically using GitHub’s suggested starter workflows from the Pages settings.

On your project repository’s home page, click the “Settings” button (appears on the right along the tabs above the code area).
On “Settings” page: click “Pages” in the left side menu.
On the “Pages” page: under “Source”, click the dropdown and select “GitHub Actions”.
Below the “Source” dropdown, a box will appear under “Use a suggested workflow” titled “Jekyll”. Click the “Configure” button.
This will open an editor page creating a new file named “.github/workflows/jekyll.yml” populated with GitHub’s starter Jekyll workflow. You can ignore the file and other options displayed on the right side, and just click the green “Start commit” button.
Fill in the commit message as usual and click the green “Commit new file” button.

Committing the action file to your repository will start the build process. It may take a few minutes for the action to complete. You can monitor the progress by looking at the status icon displayed next to your most recent commit message or by checking the “Actions” tab of your repository.

Once the action successfully completes a green checkmark will appear and your site will be live. To find the URL you can visit “Settings” > “Pages”.

Going forward, each time you push or directly commit to the repository, the action will rebuild the site.

The green checkmark will appear next to the most recent commit that triggered a successful build. If a red “X” appears next to your commit, the build failed and your updates will not be deployed–the last working version of the site will still be live. Visit the Actions tab to see detailed information about the error to help debug the issue.

Marshmallow Wreaths Recipe

Tue, 06 Dec 2022 00:00:00 +0000

My favorite Yuletime treat is marshmallow wreaths–crisp cereal embedded in sumptuous green marshmallow goo, lovingly formed into circles and adorned with red candies. Somehow, they are richer (and of course much more festive) than regular rice crispy treats.

Ingredients

1/2 c. (115g) butter
4 c. (200g) marshmallows
1/2 tsp (3ml) vanilla
4 c. (190g) cereal (corn flakes, rice crispy, or whatever)
1 tsp (5ml) green food coloring
Cinnamon Imperials / red hots (little red candies for decoration)

Preparation

Add butter and marshmallows to a large glass bowl.
Microwave in 30 second increments until fully melted, stirring occasionally.
Add vanilla and food coloring, and mix well (should be extremely green for maximum festiveness).
Add cereal and mix furiously (as your mixture is probably getting very sticky).
Painstakingly form into circular wreath shapes on wax paper or greased pan.
While still sticky, decorate each with three red hots to make pretty.

Notes

I first tried to reproduce these on my own while living in Scotland. I translated the american style measurements (cups, tea spoons, etc) into UK style metric, and found some delicious artisan marshmallows made with natural pork gelatin. I soon discovered it was rather hard and far too tedious to reproduce the wreath shapes. However, my research confirms that they are equally delicious no matter what they look like… (for example, in 2009 my efforts resulted in a large red blob, which I served to Scottish people claiming it was a famous american delicacy and received positive reviews.)

I have converted the ingredients back to american style measurements–metric is provided in parenthesis in case you bring this heirloom recipe with you in your world travels.

RAWGraphs and browser-based apps

Tue, 29 Nov 2022 00:00:00 +0000

Data Hub Tool Talk: RAWGraphs is a browser-based app for quick data visualization. With extensive help built in, this is a great tool for teaching, learning, and experimenting with charts. This session will introduce the tool, along with tips for working with data on the web and other browser-based apps. talk recording

RAWGraphs intro

https://www.rawgraphs.io/

RAWGraphs is a free, open source, browser-based data visualization app.

There are a lot of free-ish web-based data viz tools, e.g.

Google Charts (integrated with Sheets)
Florish (data storytelling)
Datawrapper (embeddable visualizations)
Tableau Public (free hosted version of famous business analytics software)

However, each of these tools are proprietary and server-based services. You need to send your data to their servers where the processing is done and your visualizations are hosted.

RAWGraphs is different –> it is a client-side JavaScript app, meaning that all the processing is done on your computer, in your browser. It works on any computer without installing anything. You don’t send any data to their server. They also don’t host your data or visualizations.

Instead RAWGraphs focuses in on being a pragmatic visual tool to efficiently move from a spreadsheet to a vector graphic visualization. Most often you would do some data wrangling (with Sheets, Excel, LibreOffice, OpenRefine) before using RAWGraphs, and maybe some vector editing after exporting your visualization (with Inkscape).

Early versions built on the svg visualization library D3.js. Part of what was cool about D3 was there was a gallery of examples with the code directly available to edit, copy, and repurpose with your own data (now closely integrated with Observable, JS browser-based code notebooks; for example of older style, see NVD3 live code). These examples make a clear connection of learning by experimenting and re-using templates. However, D3 has a steep learning curve, and you can’t just smoothly explore your data via a variety of visualizations.

As a code free alternative, RAWGraphs uses the power of JS to provide a simple-ish visual interface to create and configure data visualizations in the browser. The goal is to get you to a solid SVG export that can then be tweaked for publication and sharing.

It is a great tool for prototyping, quick exploration, and teaching/learning. For example, provide your students a link to RAWGraphs plus a link to a CSV, and have them learn about steps to a visualization–without installing anything or leaving the browser. Each chart type has a description and related tutorial, so there is a ton of learning built directly into the interface.

Walkthrough RAWGraphs app:

“Try our data samples” > Iris flowers. Check parsing options and stacking transformation utility.
select “Convex hull” (check description and tutorial)
map length to X, width to Y, Species to Groups
customize
Export as SVG

Data on the web

When trying to load data from the web you may encounter CORS errors. Browsers will not directly load files from a different domain than the page’s home domain for security reasons. However, servers can implement a CORS-enabled policy that allows cross-domain sharing–so you will have to put your data somewhere that is configured for sharing!

Here is some good options:

Publish a Google Sheet as CSV –> On the Sheet, go to File > Publish to the Web. On the popup modal, use the dropdowns in “Link” tab to select the sheet name of your metadata (usually “Sheet 1”) and “Comma-separated values (.csv)” options, then click “Publish” button. Copy the link that is provided.
Publish on GitHub –> put your data in a repository, then use the “raw” link provided. It follows the pattern: “https://raw.githubusercontent.com/username/repositoryname/branchname/filename”
Publish on GitHub Gist –> create a new Gist, paste or drag your data in, make sure you provide a correct filename. Use the “raw” link. It follows the pattern: “https://gist.githubusercontent.com/username/hash/raw/hash/filename”
Handy note: GitHub will preview render geojson on a map!

Other client-side apps

Kepler.gl (geospatial tool that can export a static web app)
mapshaper (utility for manipulating geospatial data, helpful for transforming geojason. mapshaper source code)
Datasette-lite (data exploration tool Datasette running in locally in your browser using webassembly)
Koia (more complicated viz tool)
Vega language ecosystem with online tools Lyra and Voyager (steeper learning curve)
Starboard (beta in-browser static/server-less notebook. Can also run Python Jupyter notebooks using Pyodide, demo python REPL)
flowchart creator
jsonvisio (visual json editor)
Dillinger (markdown editor)
Squoosh (image optimizer utility)
Photopea (full image editor, plus basic vector editing)
Weird text utilities: Plain Text Format, CyberChef, Word-cleaning

Data viz resources

Issues Switching NVIDIA mode on Ubuntu

Tue, 22 Nov 2022 00:00:00 +0000

Ubuntu supports NVIDIA graphics cards out of the box with a proprietary driver that allows you to switch between different modes on your laptop for battery life, called “PRIME profiles”:

nvidia (nvidia gpu always on high performance mode)
on-demand (turn on nvidia for select tasks only)
intel (use only builtin gpu for power saving)

Any change in profile requires a restart.

Pop!_OS does a good job of integrating switching modes directly in the tool bar, but Ubuntu doesn’t offer that option out of the box. In general, switching profiles is possible using the “NVIDIA X Server Settings” application, which also gives you lots of details about your graphics card when in NVIDIA modes. However, I find that the NVIDIA settings app can switch from intel into nvidia modes, but not back to intel mode…

So as an alternative that always works, you can use the command line tool prime-select to switch profiles. Use prime-select -h to list the options:

To check the profile currently in use, prime-select query.
To switch to a different profile, use sudo prime-select intel (with either nvidia, intel, or on-demand option). Then restart your system!

Baritone Ukulele stings

Fri, 01 Apr 2022 00:00:00 +0000

Finding strings you like on your ukulele is surprisingly complicated. It is a very personal choice, depending on your ukulele, the sound and feel you like, and your playing style. And it’s hard to find advice (its all conflicting forum-post rant style garbage).

I enjoyed reading this ongoing strings journal “Best Strings for Baritone Uke: A Three-Year Experiment”. So thought I would do something similar for myself. As most of this blog, this is mainly notes to my future self!

p.s. I find this west music how to replace strings helpful for knot diagrams.

Baritone Ukulele Context

I’m a intermediate beginner, not particularly good. I pick and strum, but really just like to strum out campfire jams. I play DGBE, low D. I have two cheaper end baritones:

Kala Satin Mahogany Baritone KA-B (archived) - traditional, standard cheap baritone with a nice rich tone.
Ohana Solid Spruce & Round Composite Back Baritone BK-70RB (archived) - slightly larger, slightly more guitar-y sound, crisp distinct tones. low action set up. I really like this weird, unique sounding uke!

I have purchased most strings from Strings by Mail, except Uke Logic on Etsy.

Strings I Have Tried

When I get ready to change strings, I record a video sound check of myself with a few chords and picking greensleeves on the old strings. Once the new strings are fully settled in, I record the same thing. This gives me a chance to listen to the differences. All the strings have been very different, and also distinct on each ukulele. It is hard to characterize without an extensive music vocabulary, but these are my impressions.

Organized from favorite to least favorite.

Uke Logic Baritone “Soft Tension”

These are my current favorite strings! They sound wonderful and seem extra playable.

I choose the “pink” version, which is just cosmetic, a very light pink tint to the clear fluorocarbon strings–but looks nice, interesting at least. For low D you have a choice of fluorocarbon or gold smooth wound–so far I have only used the fluorocarbon with this set (I did use the gold wound in the “Hard Tension” version, and it is a beautiful string!). The fluorocarbon D is a reasonable thickness and the feel seems reasonably balanced with the other strings. All strings easily fit standard nut.

On BK-70RB:

Crisp and clear, ring out beautifully with a rich sound. Pleasant medium tension, similar to PhD (no buzz on low action), but seem easier to play.

After playing on the “hard” version, these seem light and airy–buttery smooth and easy to play, the sound just seems to jump out. Compared to the “hard” they seem to have more sustain. The “hard” have a bit more unique sound, but I love the playability of these strings, and they sound great.

On K-B:

Immediately after restringing, the sound was a revelation! They sound really good, feel really good. Seemed to settle in quickly.

String	Material	Diameter (mm)
E	Pink Fluorocarbon	0.508
B	Pink Fluorocarbon	0.635
G	Pink Fluorocarbon	0.813
D	Pink Fluorocarbon	0.914

Uke Logic Baritone “Hard Tension”

On BK-70RB:

Seemed to settle in quickly. Good volume and sustain. Full, rich tone, mellower than other bright flourocarbon, but still has that crispness.

The gold wound string is smooth, seems very stiff and heavy when installing, but feels good and balanced with other strings when playing. It seems to have a real strong, driving sound giving body to chords without overpowering. This wound string has very good durability–it was still sounding and looking good when I switched out the fluorocarbon strings for being a bit worn out.

The tension is noticeably “hard” compared to other strings. You can really play them hard, dig into the strumming–but I didn’t realize until switching back to “soft” strings that it is more work to actually play these.

I really like the sound and feel of these strings. They seem unique.

String	Material	Diameter (mm)
E	Pink Fluorocarbon	0.584
B	Pink Fluorocarbon	0.711
G	Pink Fluorocarbon	0.914
D	Gold smooth wound	0.762

There is an option for D Flourocarbon at 1.06mm. This seemed too big for my nut, so I didn’t try it.

PhD Strings - Premium High Density

On KA-B:

After using the stretchy Worth Browns, these felt a much nicer balanced tension. They seem to have a full bright tone, with strong sustain and volume. They feel good on fingers, smooth.

PhD doesn’t publish diameters anywhere I could find. The G string seems thicker than other sets, enough to stick in the nut, requiring some extra attention to tune.

Material is “PVDF”, some sort of fluorocarbon.

Worth Brown

On KA-B:

Nice rich, mild sound. Seemed to make the KA-B’s tone crisper and louder than Aquilas.

They have an oddly low tension, or sort of stretchiness, that I didn’t really like. They took forever to settle in and stay in tune.

String	Material	Diameter (mm)
E	Brown Fluorocarbon	0.620
B	Brown Fluorocarbon	0.739
G	Brown Fluorocarbon	0.810
D	Brown Fluorocarbon	0.909

Lots of people repeat that Worth brown and clear are the same–that doesn’t seem true, Worth intends them to have different sound qualities, not just looks.

Aquila Nylgut

Came as stock on both my Baritones (which is very common).

On KA-B:

They sounded okay, nothing particularly noteworthy–flourocarbon strings sound much crisper in comparison. However, I wore out the wound strings very rapidly, leaving cuts at each fret, so I had to change them out soon. Much prefer the durability of fluorocarbon.

On BK-70RB:

They sounded very crisp, very loud, powerful, a bit jangley like a guitar–I liked the sound a lot. There seemed to be wider dynamics possible compared to other strings. The nylgut strings feel uniquely crispy and responsive. They occasionally had a buzz on the wound strings–not enough tension on low action. However, I hated the feel on the BK-70RB–the wound strings feel very different from the nylgut, and the tensions are so inconsistent. It bothered me enough to change them before wearing them out.

String	Material	Diameter (mm)
E	nylgut	0.67
B	nylgut	0.84
G	wound	0.60
D	wound	0.70

I am intrigued by other unique Aquila strings (Sugar and Red), but since they all come with the same two wound lower strings there doesn’t seem like much point to trying them out–not enough durability.

Oasis Fluorocarbon Uke Baritone UKE-8200B

Strings come as double length size, so each pack is really two sets, meaning these are actually the cheapest strings I have tried.

On BK-70RB:

First impression was the tone sounds nice, clear, and distinct. However, I noticed a serious loss of volume and sustain. The surface has an odd grippiness, which feels fine–but when I play creates terrible squichy sounds from my fingers–it drove me crazy. I also had a buzz on the G string, I guess due to lower tension on low action.

After playing these for two weeks I hated them. They felt sort of dead. Bad.

String	Material	Diameter (mm)
E	Fluorocarbon	0.620
B	Fluorocarbon	0.739
G	Fluorocarbon	0.800
D	Fluorocarbon	0.909

String Diameter table


Uke Logic Baritone Hard	E	Pink Fluorocarbon	0.584
Uke Logic Baritone Hard	B	Pink Fluorocarbon	0.711
Uke Logic Baritone Hard	G	Pink Fluorocarbon	0.914
Uke Logic Baritone Hard	D	Gold smooth wound	0.762
Uke Logic Baritone Soft	E	Pink Fluorocarbon	0.508
Uke Logic Baritone Soft	B	Pink Fluorocarbon	0.635
Uke Logic Baritone Soft	G	Pink Fluorocarbon	0.813
Uke Logic Baritone Soft	D	Pink Fluorocarbon	0.914
PhD Strings	*	PVDF fluorocarbon	none given
Worth Brown	E	Brown Fluorocarbon	0.620
Worth Brown	B	Brown Fluorocarbon	0.739
Worth Brown	G	Brown Fluorocarbon	0.810
Worth Brown	D	Brown Fluorocarbon	0.909
Aquila Nylgut	E	nylgut	0.67
Aquila Nylgut	B	nylgut	0.84
Aquila Nylgut	G	wound	0.60
Aquila Nylgut	D	wound	0.70
Oasis Fluorocarbon UKE-8200B	E	Fluorocarbon	0.620
Oasis Fluorocarbon UKE-8200B	B	Fluorocarbon	0.739
Oasis Fluorocarbon UKE-8200B	G	Fluorocarbon	0.800
Oasis Fluorocarbon UKE-8200B	D	Fluorocarbon	0.909

Paged.js for HTML to PDF

Sat, 05 Feb 2022 00:00:00 +0000

Paged.js is a javascript library to create high quality print layouts from your HTML content. It is designed to be part of a new content to print workflow that cuts out all the messing around with clunky word processors, layout design, and desktop publishing software. There are some amazing examples of print books created using it. Also check out this demo using Aurorae (text from Project Gutenberg).

I think this has a lot of potential for creating open educational resources in static web workflows.

OER tools often want to use Markdown content to output both a website and PDF version. Currently, that generally involves using Pandoc plus LaTeX to create PDFs at some point. This is limiting because controlling the layout of HTML processed through Pandoc is difficult, as well as adding extra software for users to install.

To use paged.js integrated into a static web workflow we don’t need to install anything else. We simply gather all the content on one HTML page, add the library and some custom print CSS, and then view the output in the browser. You can use dev tools to tweak the CSS to get instant feedback on your changes. Then print using the browser’s native print functionality.

Note: the fully features only seem supported in Chrome browser, and Chrome’s builtin PDF printing seems a bit better. Paged.js calls the library a “polyfill” because it is implementing the CSS “Paged Media” standards that aren’t currently supported by browsers.

Aladore Test

To play around with the possibilities, I created a print version of Aladore on my aladore-book project. aladore-book is an edited version of a public domain novel, with each chapter contained in a Markdown stub, that is generated into a website using Jekyll. Using Liquid in Jekyll, I also set up a variety of data derivatives and text outputs made available on the Aladore Data page.

Starting from the minimal HTML full text version, I added the full CSS used by the main website version, the “preview” CSS provided by pagedjs, plus a handful of custom print styles:

<link rel="stylesheet" href="/aladore-book/assets/css/main.css">
<link rel="stylesheet" href="/aladore-book/assets/pagedjs/preview.css">
<style>
    /* page basics */
    @page {
        size: letter;
        margin: 1.5in 1.5in;
    }
    /* chapter page set up */
    @page chapter {
        /* add page number at bottom */
        @bottom-center {
            content: counter(page);
        }
        /* add title header */
        @top-center {
            content: "Aladore";
        }
    }
    /* start page numbers from first chapter */
    #chapter01 { counter-reset: page 1; }
    /* set up page breaks */
    .frontmatter-pages {
        break-before: right;
    }
    .chapter-pages {
        page: chapter;
        break-before: page;
        margin-top: 8em; 
    }
    /* keep figures on page alone */
    figure {
        max-width: 100%;
        max-height: 100%;
        break-before: page;
        break-after: page;
    }

</style>

Note: using inline comments // seemed to mess up Paged.js parsing.

At the bottom of the page, we add the paged.js “polyfill”:

<script src="/aladore-book/assets/pagedjs/paged.polyfill.js"></script>

This gets us to the Aladore full text for print version. Open the page in Chrome browser, use the browser “print” option, and “save as PDF”.

Getting Started with Raspberry Pi

Thu, 03 Feb 2022 00:00:00 +0000

Curious about Raspberry Pi and other single board computers? This session will introduce everything you need to know to get started, from burning an SD card to Linux basics. Single board computers can be used to power research projects, host a local media server, or just play video games–so join us to learn about all the interesting things you can do with a Pi!

What is Raspberry Pi?

Raspberry Pi is a very popular family of single board computers (SBCs) designed to be simple, flexible, affordable, and educational. Basically, it is just a minimal computer with everything contained on a small circuit board.

It isn’t much different than the insides of your laptop:

Single board computer: 64-bit CPU, 512MB - 4GB RAM, graphics card
Connect: USB, Ethernet, Wifi, Bluetooth
Output: Video (HDMI), audio (HDMI, 3.5mm jack), GPIO pins

However, this minimal form factor, and modular input and output options (such as GPIO pins) invite us to interact with it very differently from a standard laptop. Raspberry Pi is ready to act like a typical computer, or to power anything you can dream up! It is commonly used in education, prototyping new hardware, controlling unique science experiments, and powering hobbyist computing projects.

There are other SBCs on the market, but Raspberry Pi has remained the most popular. Here are some reasons why:

High quality build with stable form factor and specs. Wide adoption and stability improves the sustainability and interoperability of your projects.
Large ecosystem of compatible “hats”, “bonnets”, and add ons. There are tons of vendors out there making things that work with Pi, which makes it easier to build your project!
Well supported and maintained software and Linux operating system. Many SBCs rely on community builds for software, which can be frustrating and unsustainable. An official Raspberry Pi OS is maintained by the foundation, and many Linux projects distribute official builds as well, providing a solid and reliable base for your project.
Excellent documentation, both official and community projects. If something goes wrong or you need to figure out how to do something, there is usually help readily available.
Extensive learning materials available, including many tutorials, courses, and teacher guides. Pi started as an educational initiative and retains that spirit. There is curriculum around learning the hardware, as well as learning more generalized concepts about computing, coding, linux, inventing, making, and more.

Raspberry Pi .com is where you can shop for hardware, download software, find forums and read documentation. The educational foundation maintains a separate site at Raspberry Pi .org where you can find teaching and learning resources (not all Pi based), and research.

Raspberry Pi vs. Arduino

Raspberry Pi is the most popular SBC, Arduino is the most popular microcontroller. Both are very flexible, with great documentation and community. Which one you need depends on your project!

Arduino:

Microcontroller board, 8-bit processor, tiny memory.
Very simple, solid, reliable.
Very low power consumption.
Doesn’t have an operating system, it just runs your code, i.e. it does one thing really well!
Generally programmed in C/C++ (not the easiest to learn), and require code to be compiled for flashing to the board.

Raspberry Pi:

Single board computer with full operating system.
Having an operating system means you can do lots more stuff, like outputting to a monitor, running software, and using high-level programming languages (mainly Python).
Relatively simple and reliable, but OS requires updates, might need restarting, etc.
Consumes much more power than a microcontroller and requires steady power supply to avoid issues.

A new ecosystem of microcontroller boards powered by MicroPython / CircuitPython is emerging which removes some of the barriers of the traditional Arduino programming. These are worth checking out if you want to use Python, or need a board slightly more powerful than Arduino but don’t need a full computer!

What Can You Do with Raspberry Pi?

Use it as a personal computer running Linux.
Set up a media center connected to your TV.
Build a high end audio player controllable by your phone.
Play games on dedicated emulator system, cabinet, or handheld.
Print poems?

To get an idea of the range of projects out there, browse adafruit tutorials, Raspberry Pi Projects, or Hackaday.

Getting Started

Hardware: Raspberry Pi Starter Kit

What you need:

Pi SBC (see spec chart). Most common models:
- Raspberry Pi 3 Model B+ - the standard for projects for awhile.
- Raspberry Pi Zero 2 W - basically a Pi 3 in tiny form factor (and cheap price)! This is great if you need to fit your project into a small space. For general prototyping, Pi 3 might be easier to work with.
- Raspberry Pi 4 Model B - newest, most advanced and powerful option, but also uses more power, creates more heat, and costs more! You can get it with up to 8GB ram, making it very viable as a desktop computer.
- Although the specs vary, the models are mostly interchangeable, so the best choice may simply be the one that you have readily available.
Power supply. Most Pi run off a micro USB charger, but you need a decent one! A rainbow icon shows up on screen if power goes low.
Micro SD card. This will be your Pi’s hard drive. Try to get decent quality, high read/write speed, “SDHC, Class 10 / UHS 3”, min 8GB. Low quality SDs are a common source of issues.

You might want:

HDMI cable to attach to monitor or TV.
USB Keyboard, mouse, or other input devices. Bluetooth also works once you get everything set up.
Console cable can be used for headless setup and prototyping.
Ribbon cable + breadboard for circuit prototyping.

Software: Getting an Operating System

Many people take operating systems for granted since they come pre-installed on Windows and Apple computers–you don’t get any choices with these proprietary systems! Raspberry Pi does not come with an operating system or even a “hard drive” for it to load from. If you just plug it in, it can’t do anything–first you have to choose and install an operating system.

Pi’s are designed for Linux, a family of open source operating systems used to run all kinds of computers, from servers to super computers. Check the Try Linux workshop for a general introduction and resources.

Linux OS are usually provided as a downloadable disk image (sometimes called ISO), that can be burned onto a SD card. The basic steps for Pi:

Choose and download an OS image.
Burn the image onto a SD card. (Keep in mind this is not the same as copying the files over!)
Insert SD into Pi.
Plug in the Pi to boot up! (Pi’s don’t have power buttons…)

The easiest way to complete these steps is using the “Raspberry Pi Imager” app. Imager will download select OS images, burn SDs, and provide other helpful utilities (such as reformating SDs). Visit the Raspberry Pi Software page, download Imager for your computer, and install.

For detailed instructions see Pi Getting Started, for the official step-by-step guide for a visual walk through.

Just keep in mind, once you burn an image, Windows won’t be able to read or use the SD. You can use Pi Imager to “Erase” the SD to restore it to normal.

You don’t have to use Imager! The official images or other ISOs can be downloaded manually, then burnt the SD using any disk imaging software, such as Etcher. Check burn ISO note for details.

Choosing an OS

Which operating system you choose depends on your use case. For the officially supported Pi OS you have two main choices:

“Raspberry Pi OS with desktop” - is the generally recommended version. It is compatible with all Pi versions. The “desktop” is the visual interface you are familiar with on Windows or Mac. This is the default choice on Imager.
“Raspberry Pi OS Lite” - does not come with a desktop. This is a good option for headless projects or servers where you don’t need to use a visual interface.

Each version also comes in 32-bit or 64-bit editions. The 32-bit versions are compatible with all Pi models and are the default. On Pi 4, the 64-bit version may provide better performance and compatibility with newer software.

In addition to the official Pi OS, many other Linux distributions provide images built for Raspberry Pi. Some of these are specialty OS, such as media servers or game consoles. Others are general purpose OS or servers from popular distros. Popular options include:

Ubuntu, one of the most popular Linux distributions for desktop and servers, so may simplify some projects or be familiar. Desktop version only runs on Pi 4.
LibreElec, a minimal OS designed to run as a media center on your TV.
RetroPie, a minimal OS designed to run game emulators on your TV.

The confusingly named “Raspberry Pi Desktop for PC and Mac” is simply an image of the OS built for normal laptop architecture. This gives you the option to run “Raspberry Pi OS” on a non-Pi computer–but you might want to check out other distros! Raspberry Pi OS is based on Debian Linux.

Boot Up!

Once you SD is burnt:

Insert SD into your Pi.
Plug in any peripherals, such as HDMI to a monitor/tv and a USB keyboard.
Plug in your Pi power supply.
Wait for it to boot!

You will see a bunch of text flash past on the screen–that is okay and normal. Eventually you will get to the desktop and you can start using your new Linux computer!

Check the official OS docs for lots of details.

Even for headless projects, it is often easiest to start with your Pi plugged into a monitor for initial set up. If your OS does not come with a desktop, you will see the text terminal interface. Alternatively, you can use a console cable or SSH into your Pi for remote access.

Git Command-line Tokens

Wed, 01 Dec 2021 00:00:00 +0000

Most git repository hosts have changed authentications methods in the last few years to increase account security. Rather than using your account password to authenticate for push/pull on the command line, you will need a special token or app password set up.

GitHub Tokens

GitHub requires using a “personal access token” rather than your account password for command-line git commands since around Aug 2021.

On GitHub click your profile image in upper right, click Settings > Developer settings > Personal access tokens. Generate a new token, check off the “repo” level of access. Copy the token to secure password manager (treat it just like a password).

When you authenticate on command-line, use username as normal, but paste in the token rather than your password.

Bitbucket App Passwords

Bitbucket requires using an “app password” rather than your account password for command-line git commands (see notice).

On Bitbucket click your profile image in lower left, click “Personal settings”, then “App passwords”. Create a new app password with the “Repository” level permissions checked (doesn’t need any of the other types of permission). Copy the password it provides (treat it just like any other password).

When you authenticate on command-line, use username as normal, but paste in the app password rather than your account password.

Update Credential Manager

Git for Windows comes with git credential manager installed and configured, so your password/token is probably automatically stored (i.e. you don’t have to type your password in for every git operation). On Linux you can also set up a Git Credential Helper.

If you need to update your credentials to a new token/password, you will need to delete or update the values stored in your credential manager.

On Windows look in Control Panel > User Accounts > Credential Manager > Windows Credentials. Then under “Generic Credentials” > Edit, look for the corresponding entry like “git:https://bitbucket.org”.

On Linux the easiest method is to use GNOME “Passwords and Keys” (i.e. Seahorse), the GUI to use libsecret. Open “Passwords and Keys”, then look in Passwords > Login category. You should find entries that correspond with the repo platforms (github.com or bitbucket.org etc). Double click the listing to open the properties. Either delete the old one (then provide updated version on next git operation on commandline)–or click in the password box, paste in the new token/password, then close the box (this will ask if you want to save).

Intro to HTML, SEO, and Analytics

Wed, 03 Nov 2021 00:00:00 +0000

workshop outline for writing for the web

HTML

Despite the slick interfaces and amazing websites out there today, everything that is displayed in your browser is made up of HTML, CSS, and JS.

(i.e. the same components that made up the web since the beginning: U of I Library website, Feb 1, 1997)

Yet, if you use a modern Content Management System such as WordPress, Drupal, SquareSpace, WIX, or Google Sites, you may have created entire sites without ever seeing HTML.

CMS are “dynamic web” applications that provide a web-based administration interface to create and manage a website. Under the hood CMS consist of software that runs on a server (server-side processing) and a database to store information. When a user visits a web page (i.e. by clicking a link), their browser sends a request to the site server –> the CMS interprets the request, retrieves the correct data and templates, and sends the user a response (i.e. HTML).

So to better understand how the web works and what your CMS is doing, it is helpful to dive into the fundamentals of HTML.

Lessons:

SEO

Search engine optimization is a sort of mad science of how to get your site ranked higher in search indexes… In early 2000’s it was essentially spammy attempts to boost apparent relevance of pages by jamming them full of keywords. Search engines are some what smarter than that now, but keep the exact details of indexing workflows and algorithms secret to avoid sites trying to game the system.

The basic idea is that search companies use “crawlers” to surf the web following links (since there isn’t some central directory of everything!) to collect content to index. These bots aren’t quite like human users and pay a lot more attention to the code than the visual elements we see.

There is no magic SEO. You can only follow the best practices to present your site in the best possible way, so that crawlers can find and correctly understand the content.

The best way to do this is to have a well organized site with well structured pages–which will also benefit your human users! While many of the technical details should be handled by your CMS, as a web designer and web writer you will still make many choices that impact your SEO. To learn more details check the Google SEO Starter Guide and Google Webmaster guidelines.

Things Google cares about when crawling and indexing:

Mobile friendliness –> use Mobile-Friendly Test to ensure Google considers your pages mobile friendly.
Page speed –> use PageSpeed Insights to see what Google crawlers think about your site performance.
Well formed HTML with descriptive title and headings –> remember those HTML fundamentals and view-source!
Sensible link structures and good navigation links. Also sitemap XML can help.
Structured markup such as Schema.org. Example, view source of pages and items in Barnard-Stockbridge digital collection.
Other sites link to your content.
People click on your pages in search results –> use Google Search Console to see how your site content looks to Google and understand what searchers are pointing people to your site.

Analytics

Once you have a site, you often want to know if anyone is visiting it –> or maybe who is using it and how they are using it! If you make changes to the navigation, features, or content, it can be helpful to have data to understand the impact of your updates. This is the value of site analytics.

There is a few main types of use data available:

Server logs –> your server records basic information based on requests sent to it. This data can be analysed to provide statistics about usage. The numbers are usually pretty accurate (it is impossible to block, but requests can fake the information). Example: logs in Reclaim host
CMS statics –> most CMS platforms will have some analytics built into the system. This is often analyzing data similar to server logs, but can be more meaningful since the CMS can connect the requests with content. These are much easier to use and interpret since they are built into the CMS admin interface. Example: WordPress stats
JS analytics –> to understand what a user actually does when they visit a page, it is necessary to use JavaScript loaded in their browser. The most widely used platform to do this is Google Analytics, which track detailed information about visitors and interactions, providing enriched data about user groups. This type of analytics can be centralized across many related sites, doesn’t require access to a server, and can be used to track links from marketing campaigns. The disadvantage is that JS analytics libraries can be blocked by users, so may be becoming less accurate (see Plausible post), and they present privacy trade-offs to your users (i.e. handing over user data to Google). Example: Google Analytics

It is important to understand that analytics numbers are squishy –> they should be used to understand trends, make comparisons, and evaluate changes, not be taken as exact statistics.

Notes:

Get started with Analytics – current version is called “GA4”. Sign in using your Google account and set up a “property” for your website.
Some hosted CMS platforms don’t allow you to add your own Analytics code when using a “free” tier account. E.g. WordPress.com requires paid account.
There are alternatives to Google Analytics! Matomo is largest open source full analytics platform that can be self hosted (free but need to maintain yourself) or paid hosting. However, there are many more alternatives popping up, with many focused on user privacy and simplicity (vs. the very complicated and heavy Google Analytics). Consider if you need all that user data at all–for many, stats built into CMS platforms is enough.

Resources

Learning and reference:

Accessibility:

Freely licensed images:

Design tips:

10 Usability Heuristics for User Interface Design - famous Nielsen Norman Group article about UX.
Writing Hyperlinks: Salient, Descriptive, Start with Keyword
Why Your Links Should Never Say “Click Here”
don’t use link shorteners
Laws of UX
Web Field Manual
Butterick’s Practical Typography
Adobe color wheel – play with color schemes and check accessibility.
Coolors

Git Credential Helper

Tue, 01 Jun 2021 00:00:00 +0000

Git for Windows comes with git credential manager installed and configured, which makes using git on command-line much easier (i.e. not having to type your password every time). (note: git credential manager is now useable on Linux too)

On Linux you can set up libsecret as a similar secure credential manager.

Configure Git as usual:

git config --global user.name "Evan Will"
git config --global user.email "[email protected]"
git config --global core.editor "nano -w"

Install dependencies:

sudo apt-get install libsecret-1-0 libsecret-1-dev

Build the helper (included with git):

cd /usr/share/doc/git/contrib/credential/libsecret
sudo make

Configure git to use the helper:

git config --global credential.helper /usr/share/doc/git/contrib/credential/libsecret/git-credential-libsecret

The next time you need a credential (e.g. pushing), it will prompt for username and password/token as normal, but it will be stored by libsecret and can be used for any future authentication to that site (e.g. github, bitbucket, etc).

Update Credentials

If you need to update your credentials, the easiest method is to use GNOME “Passwords and Keys” (i.e. Seahorse), the GUI to use libsecret. Open Seahorse, then look in Passwords > Login category. You should find an entry such as https://[email protected]/ that corresponds with the credential. I usually just delete the old one, then provide the updated version on my next git operation.

Older version: gnome-keyring

Older versions of Linux might use gnome-keyring rather than libsecret. Here is the old version:

sudo apt install libgnome-keyring-dev
cd /usr/share/doc/git/contrib/credential/gnome-keyring
sudo make
git config --global credential.helper /usr/share/doc/git/contrib/credential

Copy a Git Repo (Don't Fork)

Thu, 20 May 2021 00:00:00 +0000

“Fork” is a cool concept on GitHub, but a bit misunderstood–if you want to start your own repository (not contribute back to the project), it is not the best way to make a copy of the code.

Forks

“Fork” is a GitHub feature (not part of Git), designed for a workflow similar to using git branches to collaborate with people who do not have direct access to a repository (fork docs).

Clicking the fork button on a repo creates your own “fork”–a full copy of the repository code and history with a link configured back to the original, with the same name as the original. It is intended to be used to contribute changes back to the original via Pull Requests, thus the “fork” repository will not have the full features of normal repositories.

If you want a copy of a repository to do your own thing or to start a new project, forks are not the best solution.

Copy a Repository

If you want to make an exact copy of another project you can:

GitHub Import: On GitHub click the plus icon in upper right, select “import repository”, then paste in the clone link from the repo you want to copy. The new repo will have the full code and full history of the old one–but will not be configured as a fork and will have no link to the original. It will be a fully featured repository.
Copy files: Always keep in mind a GitHub repository is just a folder of files… Create a new empty repository in your account. Add all the files from the other repository. The new repo will have the full code, but not the history of the old one. Add contents via:
- on old repository, click “Code” button and select “download zip”. Extract the zip and copy the files to your new repo (note: the download zip will not have files such as .gitignore or other workflow files that you might want!).
- clone old repository, make sure you have “view hidden files” enabled, then copy all the files except “.git” from the old repo folder and paste into the new one.
GitHub Template: if a repository is configured as a “template”, you can simply click the green “Use this template” button near the “Code” button. This will start a create a new repository with the full code of the main branch, without the full history. The new repo will display a “generated from…” link on its homepage, but there is no direct link configured between the repositories.This is the best way to copy a repository if it is configured.
- if you own a repository you want to repeatedly use a model/template for others or yourself, look in the “Settings” tab towards the top for the check box “Template repository”. Check the box.

Convert Videos with FFmpeg

Sat, 01 May 2021 00:00:00 +0000

FFmpeg is a powerful command line tool to work with video files. Video formats are complex–they involve combinations of video encoding, audio encoding, and containers–it all gets really confusing fast… But FFmpeg will help you deal with conversions pretty quickly.

Install FFmpeg

Windows install with downloaded installer: https://www.ffmpeg.org/
Mac install using Homebrew: brew install ffmpeg
Ubuntu install from repository: sudo apt install ffmpeg

FFmpeg is a command line application, so to use it:

open your terminal and navigate to the directory containing your videos (or open the terminal in that folder!)
type your FFmpeg commands, starting with ffmpeg

For full information check FFmpeg docs.

Convert AVI to MP4

To losslessly copy your AVI video into the MP4 container (i.e. without re-encoding the video stream), use the command:

ffmpeg -i input-video.avi -c:v copy -c:a copy -y output-video.mp4

This command uses the options -c:v copy and -c:a copy to copy the video and audio streams without changing the encoding into the new MP4 container.

If you want to do a whole folder of AVI videos, you can use a Bash loop, like:

for f in *.avi; do ffmpeg -i "$f" -c:v copy -c:a copy -y "${f%.avi}".mp4; done

This loop goes through all AVI files in the current folder (*.avi), copies each one to a new MP4 file, and names it using the same base filename swapping out the extension ("${f%.avi}".mp4 removes “.avi” and adds “.mp4” to the filename).

If you get an error using the copy method, it probably means your AVI contains encodings that are not compatible with MP4. To just get it to work with the default options, just remove the -c flags, using just:

ffmpeg -i input-video.avi -y output-video.mp4

for f in *.avi; do ffmpeg -i "$f" -y "${f%.avi}".mp4; done

This might not be the optimal conversion, but it should work!

If you want to take more care, use ffprobe -i input-video.avi to learn more about the video and audio encodings embedded in your AVI. Then use specific encodings in the -c flags to control your output file.

Compare Matomo to Google Analytics

Mon, 01 Mar 2021 00:00:00 +0000

this is a draft page to peek at the data. I hope to add more data and better viz.

Adding Google Analytics tracking code to your site has been standard practice for years to gather data about visitors. However, many people are more critically considering this practice–Analytics is “free” because you are giving your user’s data/privacy away to Google in return. And with growing numbers of users blocking analytics, your data might not even be very accurate (see Marko Saric 1 and Marko Saric 2).

Luckily, there are a growing number of alternative options out there.

One fairly easy to adopt option is Matomo, an open source analytics platform. It is available as a one-click install on many hosts.

From Dec 2020 until Mar 2021 I had both Google Analytics and Matomo on some of my sites. This note compares the data from the two services.

Set Up

At bottom of head section:

Matomo code came first loaded from a personal .org domain, self hosted on Reclaim.
Google Analytics loaded via analytics.js version.

I collected the data for a set of top pages that have both analytics.

Pageviews

Page	Month	GA views	MA views	Difference
Setup Jupyter with Py 2, 3, and R	2020-12	678	618	8%
Transfer iPhone to Linux	2020-12	325	291	10%
Add more to Git Bash on Windows	2020-12	111	85	23%
Go-Go GH-Pages home	2020-12	189	140	25%
Setup Jupyter with Py 2, 3, and R	2021-01	680	623	8%
Transfer iPhone to Linux	2021-01	429	344	19%
Add more to Git Bash on Windows	2021-01	137	120	12%
Go-Go GH-Pages home	2021-01	190	187	1%
Setup Jupyter with Py 2, 3, and R	2021-02	693	608	12%
Transfer iPhone to Linux	2021-02	297	266	10%
Add more to Git Bash on Windows	2021-02	177	152	14%
Go-Go GH-Pages home	2021-02	107	93	13%

GA seems to consistently record more pageviews.

In 2021-01 Google Analytics on sites across the world were hit by a spam spike from bot-traffic.icu (in my case, 5040 views in a day). Matomo did not record this traffic. Does Matomo do better at filtering out junk traffic?

Unique Pageviews

Page	Month	GA uniq views	MA uniq views	Difference
Setup Jupyter with Py 2, 3, and R	2020-12	558	502	10%
Transfer iPhone to Linux	2020-12	300	269	10%
Add more to Git Bash on Windows	2020-12	106	83	21%
Go-Go GH-Pages home	2020-12	105	88	16%
Setup Jupyter with Py 2, 3, and R	2021-01	570	527	7%
Transfer iPhone to Linux	2021-01	392	313	20%
Add more to Git Bash on Windows	2021-01	129	114	11%
Go-Go GH-Pages home	2021-01	125	122	2%
Setup Jupyter with Py 2, 3, and R	2021-02	571	501	12%
Transfer iPhone to Linux	2021-02	266	238	10%
Add more to Git Bash on Windows	2021-02	160	139	13%
Go-Go GH-Pages home	2021-02	78	67	14%

GA seems to consistently record more unique pageviews.

Average Load Time

Page	Month	GA average	MA average	Difference
Setup Jupyter with Py 2, 3, and R	2020-12	2.63	1.75	0.88
Transfer iPhone to Linux	2020-12	0	0.81	-0.81
Add more to Git Bash on Windows	2020-12	2.35	2.07	0.28
Go-Go GH-Pages home	2020-12	0	1.81	-1.81
Setup Jupyter with Py 2, 3, and R	2021-01	0.76	1.81	-1.05
Transfer iPhone to Linux	2021-01	2.04	0.7	1.34
Add more to Git Bash on Windows	2021-01	0	1.81	-1.81
Go-Go GH-Pages home	2021-01	1.15	1.88	-0.73
Setup Jupyter with Py 2, 3, and R	2021-02	2.25	1.78	0.47
Transfer iPhone to Linux	2021-02	2.13	0.98	1.15
Add more to Git Bash on Windows	2021-02	1.9	0.7	1.2
Go-Go GH-Pages home	2021-02	1.63	1.34	0.29

GA load times seem more random. Not sure why they would report “0” and then rather long load times.

Browser

Browser	GA User %	MA User %	Difference
Chrome	71.88	64.5	7.38
Firefox	12.51	18.6	-6.09
Safari	7.68	10.8	-3.12
Edge	3.58	3.5	0.08

Not including other browsers (so it doesn’t add up to 100%). MA provides stats for desktop vs mobile browsers separately, so I added them together.

OS

OS	GA User %	MA User %	Difference
Windows	45.88	45.4	0.48
GNU/Linux	17.27	24.4	-7.13
Mac	15.4	18.4	-3.0
iOS	4	6.4	-2.4
Android	15.4	5	10.4
Chrome OS	1.42	0.3	1.12

The large differences here are interesting. Has MA classified some Android as Linux? Do Linux users block GA but not MA?

Other notes

Overall, MA’s interface is very nice and less confusing than GA. It is definitely a viable alternative.

Software Snapshot

Fri, 05 Feb 2021 00:00:00 +0000

I want to capture an annual snapshot of software I am using on my computer. This info is sort of embedded in other notes where I talk about setting up Ubuntu, but I thought it would be interesting to regularly record a snapshot to compare over time. I have noticed a significant decrease in software I actually use, moving more to either browser or text editor.

Jan 2021

Distro: Ubuntu 20.04 LTS - I used to really enjoy installing the latest releases and completely wiping my laptop every 6 months, but lately LTS seems less work…
Desktop:
- GNOME 3.36.8
- Extensions: Caffeine
- GNOME Tweaks
Browsers:
- Firefox 85 - I use firefox for every thing. Interesting to have work access to office365, Word is just as terrible in browser as on desktop.
  - uBlock Origin
  - Multi-Account Containers
  - Facebook Container
- Chromium 88 (snap) - I use chromium just for testing web development. I prefer the dev tools to Firefox’s.
Text editors:
- VS Code - used constantly, but getting more annoyed by each update. It seems like it is getting constantly more features, with debugging and information popping up everywhere. I don’t need or appreciate that stuff. I spend a lot of time trying to turn it off, but seems harder…
- TextEditor / Gedit - used as extra scratch pad constantly.
- Atom - installed, but not used. I want to like it (it seems like it should be slightly better than VS Code), but just seems slow and not quite right. Not having a good integrated terminal is a big draw back for me.
Other GUI software:
- GIMP
- OBS Studio
- VidCutter
- KeePassXC
- Etcher
- OpenRefine
- LibreOffice Calc
Pandemic work from home has introduced some new every day tools:
- Zoom - not great on Linux, has less features, freezes up on screen share.
- Cisco VPN
- Remmina
Command line tools:
- git
- ImageMagick
- ExifTool (via libimage-exiftool-perl)
- ffmpeg
- Pandoc
- LaTeX (via texlive texlive texlive-fonts-extra texlive-xetex texlive-extra-utils) - mainly just for use with Pandoc.
- Ruby (via rbenv) - used to do more with Python, but recently just using commandline, OpenRefine, or Ruby + Rake for anything.
- Jekyll (via Ruby)
- Java (via default-jre) - just for use with OpenRefine.
- tlp tlp-rdw - laptop battery utilities.

Deploy a Jekyll Site with GitHub Actions (v2)

Fri, 05 Feb 2021 00:00:00 +0000

This page describes manually creating an Action and was updated in 2023 to fix issues that broke with updates at GitHub. Recent updates allow you to easily use a default starter template–check Deploy a Jekyll Site with GitHub Actions using Starter Workflow for details!

GitHub Actions are a newish feature of the platform that allows you to create / run workflow automation scripts in repositories. It functions similar to 3rd party continuous integration tools like Travis CI, but built directly into GitHub. Explore the documentation and marketplace to get the idea of what you can do.

In my use case, I just want to build a Jekyll site and deploy on GitHub Pages. However, the default Pages build uses an old version of Jekyll and does not allow plugins (see dependency versions for details). Actions allows us to set up our own build instead.

Jekyll docs provide a GitHub Actions tutorial that is a bit out of date. I assume you already have a Jekyll project set up and are ready to deploy it on GitHub Pages.

Here are the steps to set up a new Jekyll build Action:

url and baseurl

If you use the Liquid filter absolute_url ensure your “_config.yml” has the url and baseurl values set correctly for hosting on GitHub Pages following the pattern url: https://username.github.io and baseurl: /repository-name.

Gemfile and Gemfile.lock

Your Jekyll project repository needs a “Gemfile” committed. I often .gitignore these since my projects aren’t complex, and they tend to confuse others trying to re-use my repos. You don’t need anything fancy. A Gemfile with just this will work:

source 'https://rubygems.org'

gem 'jekyll'

Optionally, you can also commit your “Gemfile.lock” to reproduce the environment exactly as you use it on your local computer (in theory).

Add Action YAML

Next, you need to create the YAML Action file in the directory “.github/workflows/”. You can name it what ever you want, but I call mine, “.github/workflows/jekyll.yml”.

Adding this file will automatically set up a GITHUB_TOKEN secret, which is nice so you don’t need to mess around with personal access tokens. However, you do need permission to create a “workflow” level token when committing the file. So depending on how your local GitHub authentication is set up, you might not be able to push this file or if you are editing an existing one your action might end up with permissions errors!

To avoid issues, I create and edit the file only using the web interface. On the repository home page, click “Add file” > “Create new file”. In the filename field start typing .github/workflows/jekyll.yml. This will create a folder “.github” (with a period in front!), with “workflows” folder inside, with a file “jekyll.yml” inside.

Add this text to your yml action:

name: build site with jekyll and deploy on github pages

# runs when you push or merge PR to main branch
on:
  push: 
    branches: 
      - main
  pull_request:
    branches: 
      - main

# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
permissions:
  contents: write
  pages: write
  id-token: write

jobs:
  jekyll:
    runs-on: ubuntu-latest 
    steps:
      # checkout code
      - uses: actions/checkout@v3

      # Use ruby/setup-ruby to shorten build times
      # https://github.com/ruby/setup-ruby
      - uses: ruby/setup-ruby@v1
        with:
          ruby-version: 3.1 # Not needed with a .ruby-version file
          bundler-cache: true # runs 'bundle install' and caches installed gems automatically

      # use jekyll-action-ts to build
      # https://github.com/limjh16/jekyll-action-ts
      - uses: limjh16/jekyll-action-ts@v2
        with:
          enable_cache: true

      # use actions-gh-pages to deploy
      # https://github.com/peaceiris/actions-gh-pages
      - name: Deploy
        uses: peaceiris/actions-gh-pages@v3
        with:
          # GITHUB_TOKEN secret is set up automatically
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_dir: ./_site

The on key says to build on any push or PR to the main branch (you could switch it to what ever branch works for you, just don’t try to use gh-pages branch).

The permissions key sets the access for the GITHUB_TOKEN necessary to add changes to your repository.

The jobs key gives the list of things to do.

Each uses value is a repository on GitHub, so you can go look at the code to see what it is doing, or set up your own version. In this workflow:

actions/checkout@v3 checks out the code from the main branch (from GitHub).
ruby/setup-ruby@v1 uses a pre-built Ruby and cached gems to speed up built time (when using Ruby, this is apparently preferable to using “actions/cache”).
limjh16/jekyll-action-ts@v2 runs the jekyll build (supposedly faster than “helaili/jekyll-action”).
peaceiris/actions-gh-pages@v3 takes the output and commits it to the gh-pages branch.

Once committed, the action will run each time you push a new commit or merge a pull request.

However, the first time you have to manually activate GitHub Pages. Go to the repository Settings, click “Pages” in the side nav. In the “Source” dropdown select “Deploy from a branch”, and in “Branch” dropdown select “gh-pages”, and save. After Pages is activated, you may need to create another new commit to finally get the web site live.

Note, in the “Pages” settings do NOT use the “GitHub Actions” option under “Source”. This older style Action commits the output of your build to the gh-pages branch, rather than delivering it directly to Pages. So although you are using an Action, Pages is still “Deploy from a branch”!

The Actions tab of the repository provides detailed progress (and error messages) for your workflow, so if something goes wrong it is easier to debug than default GitHub Pages.

OBS: Creating Screencast Videos

Fri, 04 Dec 2020 00:00:00 +0000

Open Broadcast Studio (OBS) is an open source, cross platform tool for recording and live streaming video. It is focused on performance for video game streaming, but works well for creating instructional videos where you would like to share a screen cast of using software.

To install, grab the package from the main website. On Linux, first make sure you have ffmpeg installed, sudo apt install ffmpeg, then add the PPA.

Tips

By default all audio sources are active which might result in crummy sound–so you might want to mute/disable desktop audio, and the mic on your webcam if you have a separate headset mic.
If you want to use a webcam in multiple scenes: first make a scene that is just the webcam full screen, then add that scene to other scenes (adding it as a source to multiple scenes doesn’t work).

Handy Utilities

vidcutter is a handy visual utility for cutting and combining videos. Installing via Flathub seems to work well–don’t use old distro repository versions, since they probably won’t work.
Use ffmpeg to convert formats: ffmpeg -i input.mp4 output.webm (see tips)
combine videos with ffmpeg

File naming for web and Git collaboration

Tue, 01 Dec 2020 00:00:00 +0000

Be extra careful of file naming case sensitivity when working with people across Linux, Mac, and Windows.

In Linux file systems, filenames are case sensitive. So files like “README.md”, “readme.md”, and “ReadMe.md” are three totally different files.

However, Windows and Mac file systems are by default case insensitive. On Window and Mac files like “README.md”, “readme.md”, and “ReadMe.md” can not exist at the same time, and all would refer to a single file.

Adding to confusion, Windows and Mac are still case preserving, i.e. the filesystem stores the original case information (“README.md” vs. “readme.md”) even though they treat filename as case insensitive. Windows Explorer lets you edit the filename case including extension. However, Mac doesn’t seem to really let you change case in the filesystem and records extensions as upper case by default. This can lead to a lot of confusion trying to fix case sensitivity in filenames and extensions.

Importantly, even if you do not have a desktop Linux user in your team, case sensitivity still matters since Git platforms, testing, continuous integration, and web servers are most likely Linux systems.

So to avoid issues:

Set up a project filename convention (probably all lowercase if a web project)
When working on Linux or web platforms don’t create files that would conflict if insensitive
When working on Windows and Mac, use actual filenames and don’t rely on case insensitivity when creating links or scripts.
Tell everyone that file naming is super important!

Issues On Web

For example, when Windows and Mac users run a local dev server (such as using Jekyll), the localhost urls and filesystem will be case insensitive. You may have a file like “assets/ExampleImg.PNG”, and the urls “localhost:4000//assets/ExampleImg.PNG”, “localhost:4000//assets/ExampleImg.png”, and “localhost:4000//assets/exampleimg.png” will work. They test using the site and everything seems to be working fine.

However, once deployed on most Linux servers, only the case sensitive filename “/assets/ExampleImg.PNG” will work. So they end up with mysterious errors in production.

Issues on Git

Git is developed and designed for Linux, thus treat files and their history as case sensitive. This can lead to nasty Git errors on Windows and Mac if a Linux user commits files that would be the same if case insensitive.

E.g. Linux user commits “README.md” in a directory that already contains “readme.md”, this is valid on Linux and GitHub. However, if Windows and Mac users pull that update they will end up with fatal errors and a lot of confusion. It can’t be gracefully fixed on Windows or Mac, so it is best to clean it up on GitHub or Linux, then do a fresh clone (see xkcd git).

Getting iPhone HEIC Images on Linux

Thu, 05 Nov 2020 00:00:00 +0000

Apple seems to spend a lot of effort trying to make their ecosystem incompatible with others. Which is frustrating, selfish, greedy, stupid, etc.

But sometimes you need to get images off an iphone. Support to just plug it in seems to go in and out on mainstream Linux (see my earlier notes), but right now it sort of works. I can plug in an iphone, tell the phone to “trust” the computer, and automatically mount the iphone’s “photos” and “documents” without too much trouble. If it doesn’t work after you “trust”, unplug and plug it in again.

Keep Originals

Once connected you will still get endless errors trying to transfer more than a few photos. It seems like the iphone in its wisdom is trying to convert the HEIC format images in its storage on the fly to JPG for you–and this kind gesture seems to cause it to timeout constantly. To fix this, you need to go into Settings > Photos > “Transfer to mac or pc” and switch from “Automatic” to “Keep Originals”. (Notice the “mac or pc” snub? Apple seems to really hate linux…)

If you are making large transfers, you still may need to unplug and plug in again a few times if you start getting errors.

Convert HEIC?

Now you have a folder of .HEIC images (HEIF format).

To decode HEIC files you need libheif, which is already installed on Ubuntu (package libheif1). However, most programs (other than GIMP) need some additional packages installed to actually start using HEIF. To use ImageViewer you need heif-gdk-pixbuf, to get thumbnails in Files you need heif-thumbnailer, and to do conversions you need extra utilities provided in libheif-examples (unless you want to manually recompile imagemagick). So:

sudo apt install libheif1 heif-gdk-pixbuf heif-thumbnailer libheif-examples

should get everything you need. Right click on a .heic image, select open with ImageViewer, and from then on they will be viewable just like any other image.

However, there isn’t a ton of support for .heic. If you are uploading images to a website or working with many apps you may need to convert them all to .jpg. GIMP can do an individual image effectively, however for batch conversion libheif-examples provides the commandline tool heif-convert. Use a bash loop in the folder of .heic images:

for f in *.HEIC; do heif-convert "$f" "${f%.HEIC}.jpg"; done

By default heif-convert seems to use “90” quality. You can set your own quality (0-100) by adding -q flag, e.g. heif-convert -q 98 input.heic output.jpg

Have fun!

Library Research and Citation Tools

Sun, 25 Oct 2020 00:00:00 +0000

Notes for graduate student meeting introducing library research and citation tools.

Library Reminders

Get help:

Reference Desk (Ask a reference librarian: chat, txt, email, don’t be shy! Hours)
Library website! (there is a ton of information and resources here)
Research Refresher video , handout (be sure to check out the full Graduate Essentials Series)

Resources:

Summit (network of 37 NW academic libraries in Orbis Cascade Alliance) and ILL (worldwide network for library sharing)
Document Delivery / digitization requests, Curbside Delivery (new no contact access methods)
Reserves (save your students money!)
Book purchase requests (we have endowed funds to buy stuff, use the form or email me)
VIVO U of I Research Network (explore research on campus)

Research Tools

Important research concepts for scholarly literature:

Citation guide, Excelsior Online Writing Lab Citation & Documentation
Peer Review
Grey Lit
Preprint archives, e.g. arXiv or engrXiv

General search strategies:

Booleans (AND, OR, NOT - learn how to expand, narrow, and refine your search terms)
Keywords and concept mapping (from your known topics explore broader, narrower, related, synonyms / thesaurus. concept mapping intro video)
Citation chaining (look at what an important paper in your area cites and who cites it)

Library Catalog (Primo)

Primo Discovery Service

Content: everything that the library has access to (books, journals, gov docs, articles, physical and digital)
Less focused and powerful than databases
Use notes:
- sign in (access resources off campus)
- account (check loan status, saved items/searches/history, requests)
- scopes (UI Library, Summit, E-Resources)
- filters (date, resource type, etc)
- “personalize” (weights search results in your discipline)
- item features (citation, permalink, email, summit libraries, virtual browse)

Library “Databases”

Database A-Z List (all databases that the Library subscribes to)

Content: each database has a specialized set of materials with enhanced metadata to aid in discovery.
Use notes:
- databases may not include full text access
- each may have different limitations and restrictions
- use proxy to access from off campus

Important ENGR databases:

IEEE Xplore (standards, personalized account)
Engineering Village / Compendex
Materials Science & Engineering Collection
ACM Digital Library (conference proceedings)
Dissertations & Theses PQDT (ETD database for most universities)
Safari: O’Reilly Learning Platform (handy tech learning resources)
SciFinder (patents, chemical abstracts)

External Indexes

Web of Science
- Content: wide coverage of science scholarly/peer-reviewed articles from the most important journals and conference proceedings. Index only (metadata, not full text), contains items that the Library might not have access to.
- Excellent citation tracking, enables in depth citation chaining
- Filter by highly cited, “hot in the field”, etc., to explore subject area
- Export data, citation lists
- WoS Search intro video, quick ref
Google Scholar
- Content: good content coverage of academic literature, but not all peer-reviewed and sources are not always clearly labeled. Index only (metadata, not full text), contains items that the Library might not have access to.
- Familiar / easy to search
- Good citation tracking
- Connect to UI Library:
  - If you connect U of I Library to your Scholar account, search results will show a link “Univ of Idaho - Get it” to access full text items in our subscriptions.
  - How to connect to UI Library
Article Access and Discovery Tools
- Variety of plugins and tools to help find access to articles that the library doesn’t have access to, or to discover more research.

Citation Management

Citation management software are tools to store citations and automatically generate bibliographies. They can help you keep track of your research, discover new items, collaborate on writing, and share resource lists for teaching/research. Start using one now, it will make your future life so much easier, saving you time and effort!

Common features:

Store references, generate any style citation (you don’t need to learn the citation styles, or convert between them!)
Organize your research (keep all your references and notes in one place)
Create groups for sharing and collaborating on research (easiest if your collaborators use the same platform…)
Auto extract citations from web pages and PDFs
Store / manage article PDFs
Integrate with word processors (MS Word, Libre Office, Google Docs)
Standalone app + web version + browser plugins

Summary of the biggest platforms:

	Zotero	EndNote	Mendeley
Cost	Free, open source, $ storage upgrades	$$$	Freemium, $ storage upgrades
Developer	Center for History and New Media/George Mason University	Clarivate Analytics	Elsevier
Pros	ease of use, availability	integration with Web of Science	integration with Elsevier

Zotero

Zotero is free, open source, cross platform, and probably the easiest citation manager to get started with. The web interface makes it easy to collaborate on and share reference lists. It has many options to import / export other citation management lists and formats. Developed by an academic non-profit, Zotero is independent and does not sell your data.

Get started

Zotero web account - first set up an account to use web version. This is important for syncing your local data, working on multiple computers, and collaborating with others.
Zotero Desktop - optionally download and install the “personal research assistant” / “stand alone” app. The installation will likely prompt you to install plugins for LibreOffice / MS Word, the “Zotero Connector”, and set up an online account.
- If you already set up your online account, add it in Preferences > Sync. Once set up, the green refresh arrow appears in upper right of the app window.
“Zotero Connector” browser plugin (Addon/Extension) - install the plugin for your web browser. This will allow you to save references / pdfs directly from web pages.
- Access Connector preferences by right clicking the icon. Check for defaults about saving PDFs and website snapshots. You may also need to “Authorize” to save items directly to the web version if your desktop app isn not open.
- Will automatically detect Proxy links (i.e. logging in via U of I Library) and offer to save that setting to facilitate access.
- Provides Google Doc integration.

Create citation records

Create a new record - use one of the methods to add references to your library:
- “Magic wand button” - paste in a URL, DOI, ISBN. Zotero will auto extract the information to create a new record. It will automatically attempt to download related PDF or create a website snapshot.
- “Plus button” - manually fill in the information.
- Click the browser plugin in your address bar - information will be auto extracted to create new record.
Edit & Annotate record - check the record “tabs” for options. Use these fields to organize and enrich your collections:
- “Info” - manually edit the citation details if necessary.
- “Notes” - your annotations about the record. You can add multiple writing in rich text. Write summaries, key points you want to use, facts, sentences to quote?
- “Tags” - add tags to group your articles around topics (see the tag visualization in lower left).
- “Attachments” - add files or links, or view current. “Snapshots” are locally saved version of a web page.
Create folders to keep organized.
Create group libraries to collaborate - setting up groups is done in the web account only (but will sync to desktop).
- share PDFs, which can be a huge time saver and simplify access.
- can be public or private. Great way to publicly share reading lists.

Generate bibliography

“Create citation/bibliography” - select some references, click the button / right click, and copy the bibliography into your document.
Word Processor Plugins - Integrate directly with LibreOffice, Word, or Google Docs. Use the tool to insert citations from your manager directly into the paper. This will add the correct inline/footnote citation and populate the bibliography/works cited section. (note: these are automatically installed by the desktop app)
RTF scan - alternative to using the word processor plugins when writing in .RTF format. Add name+date to text, Zotero scans it and matches to your records to auto generate citation and bibliography.
“Export” options include BibTeX if integrating with LaTeX writing workflows.
“Generate report” option on desktop outputs publication info plus your notes, a nice way to create an annotated bibliography to share.

Usage tips

ZoteroBib - handy tool to create a bibliography, after you skipped using a citation manager.
Zotero Storage - 300 MB for free, 2GB for $20/yr, 6GB for $60/yr, Unlimited for $120/yr. Groups storage counts against the owner’s account, so be strategic if you plan on using it extensively (i.e. have lab leader buy it so you can share all PDFs!).
Create CV

Resources:

Zotero Docs - detailed and helpful documentation.
U of I Zotero workshop video - 30 minute intro to Zotero.

DIY air filter for fire season

Mon, 14 Sep 2020 00:00:00 +0000

Record fire conditions in California, Oregon, and Washington in Sep 2020 plunged all of the Northwest in choking, hazardous levels of smokey haze. For air quality details see AirNow and the Fire and Smoke Map. There isn’t much you can do to escape. Official advice is basically just stay inside, turn your air conditioner on recirculate, and run your air purifier full time.

What if you don’t have a HVAC system, recirculate option, or air purifier?

Basically its miserable… but here is a viable DIY option to clean your indoor air made with a box fan, furnace filter, and duct tape.

p.s. more recently it has been discovered that running this type of filtration also helps protect from Covid in enclosed spaces! If you have four air filters available, build a more advanced Corsi-Rosenthal box (or collection of DIY resources from Clean Air Crew). UC Davis has published some research confirming the effectiveness of these DIY filters.

Step 1: Get Fan

Do you have a box fan? Great. Measure it, usually they are about 20” x 20” square.

Step 2: Get Filter

Wild fire smoke produces fine particle pollution (2.5 micrometers and smaller), so you need a high quality furnace filter, something like Filtrete Healthy Living that uses electrostatic charge to capture particles. Look for a MERV rating of at least 11, MERV 12 or higher is better.

If you are in a pinch and not using your furnace, you can just grab it out of your system.

If you go to the store, look for a size that will cover your box fan. Ideally, it would be 20”x20”x4” size fitting perfectly over your fan. Deeper filters provide better flow and last longer, but the standard 1” will be fine too.

However, that ideal size is rare and if you are in a smokey area filters sell out fast–so just find anything that covers the fan with the highest MERV rating. I find that odd sizes are usually cheaper than common ones, so I used 20x30x1 because it was $5 cheaper than 20x20 for the same quality!

Step 3: Duck Tape!

The filter will have an arrow showing which way the air should flow through. Lay the filter onto the back of your box fan with the arrow pointing into the fan. Use a few pieces of duct tape to secure it directly to the fan.

Step 4: Use It

Obviously. The picture below shows a fresh filter next to one used for 24hr during horrible smokey conditions.

Another tip: get a spray bottle with clean water and spritz the air around your house every so often. Moisture particles help pull smoke pollution out of the air.

Estimating Ideal Text Body Width

Tue, 18 Aug 2020 00:00:00 +0000

Typographers have a variety of stylistic and idealistic reasons for finding an ideal line length for printed text (for example see Bringhurst The Elements of Typographic Style. On the web, line length is also important for useability–too often web pages present really long lines of text across your screen, making it hard to read.

Setting ideal line lengths can be complicated to translate to the web because of all the variables of text layout. It seems like you should just be able to say max-width: 66rem;, but that doesn’t translate correctly and would be much wider than 66 characters.

Butterick Practical Typography says to use “average line length of 45 – 90 characters, including spaces”, or “to fit between two and three alphabets on a line”. Butterick’s site uses 2.31 alphabets, which in his font + spacing = 652px width for his text block layout. The idea of alphabets on a line seems like the most practical method to estimate line lengths on a website.

Two alphabets:

abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz

2.31 alphabets:

abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefgh

Three alphabets:

abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz

Find Your Ideal Line Length

put two or three alphabets in a paragraph your main body text.
viewing the page, right click the paragraph and inspect element.
using your dev tools inspector, set the paragraph to display: inline-block; (this allows you to get the size of just the text, not the full p element)
check dev tools Layout Box Model / Computed to find the calculated width in pixels.
Adjust your text block width to your ideal width (some where between the pixel length of two or three alphabets).
Enjoy your perfect typography.

This site’s text block is currently wider than three alphabets…

abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz

I like to think that is because it is mostly about sharing code blocks, which are annoying to look at if too narrow. My web book editions, like aladore-book, use a more ideal width.

Deploy a Jekyll Site with GitHub Actions

Wed, 05 Aug 2020 00:00:00 +0000

Note: GitHub Actions has become much easier than when this post was written, please see my most recent version Actions using Starter Workflow! I would strongly suggest using it instead!

GitHub Actions are a fairly new feature of the platform that allows you to create / run workflow automation scripts in repositories. It functions similar to 3rd party continuous integration tools like Travis CI, but built directly into GitHub. It is currently being pushed pretty heavily, with the new “Actions” tab added to every repository interface. Explore the documentation and marketplace to get the idea of what you can do.

In my use case, I just want to build a Jekyll site and deploy on GitHub Pages. However, the default Pages build uses an old version of Jekyll and does not allow plugins. Actions allows us to set up our own build instead.

Jekyll docs provide a GitHub Actions tutorial. However, I assume you already have a Jekyll project set up, and I made a few tweaks to their set up.

Here are the steps to set up a new Jekyll build Action:

1. Gemfile and Gemfile.lock

First, your Jekyll project repository needs a Gemfile and Gemfile.lock committed. I often .gitignore these since my projects aren’t complex, and they tend to confuse others trying to re-use my repos. However, you need the Gemfile.lock to tell the build process exactly what Gems to use–this way, if it works on your computer, it will work on the virtual machine. (Keep in mind, the VM is Ubuntu, so not sure if that is true if you use Windows…)

You don’t need anything fancy. A Gemfile with just this will work:

source 'https://rubygems.org'

gem 'jekyll'

Create the Gemfile, then run jekyll s to generate the Gemfile.lock, and commit them both.

2. Action YML

Next, you need to create the YAML Action file in the directory .github/workflows/. You can name it what ever you want, but I call mine, .github/workflows/jekyll.yml, and added this text:

name: build with jekyll and deploy on github pages

on:
  push: 
    branches: 
      - main
  pull_request:
    branches: 
      - main

jobs:
  jekyll:
    runs-on: ubuntu-latest
    steps:
      # checkout code
      - uses: actions/checkout@v2

      # Use GitHub Actions' cache to shorten build times and decrease load on servers
      - uses: actions/cache@v1
        with:
          path: vendor/bundle
          key: ${{ runner.os }}-gems-${{ hashFiles('**/Gemfile.lock') }}
          restore-keys: |
            ${{ runner.os }}-gems-

      # use jekyll action from jekyll docs example
      - uses:  helaili/[email protected]
        env:
          JEKYLL_PAT: ${{ secrets.JEKYLL_PAT }}

The on key says to build on any push or PR to the main branch (you could switch it to what ever branch works for you, just don’t try to use gh-pages branch).

The jobs key gives the list of things to do. Each uses value is a repository on GitHub, so you can go look at the code to see what it is doing, or set up your own version. In this workflow:

actions/checkout@v2 checks out the code from the main branch (from GitHub).
actions/cache@v1 checks to see if the gems listed in your Gemfile.lock are already cached (from GitHub).
helaili/[email protected] basically runs bundle exec jekyll build, then commits the output into the gh-pages branch, using the secret called “JEKYLL_PAT”.

Commit your Action in your project repository.

3. Access Token and Secret

To allow the Action to do stuff to your repository, you need to set up a personal access token and secret for it to use.

First, create a new personal access token: https://github.com/settings/tokens

I make mine specific to the project so I can easily revoke them when messing around. If you are using a public repo, you only have to give it access to public_repo, and probably don’t want to give any more. Copy the key, since you can only see it once.

Next, go to your project repository > Settings > Secrets. Click “New secret”, name it exactly JEKYLL_PAT, and paste in your key. (note: “JEKYLL_PAT” is set in the jekyll.yml workflow, so if you really wanted you could name it something else if you edit it there).

4. Commit / PR

With everything set up, the Acton should run on your next commit or PR to main. You do not need to activate GitHub Pages in settings, it will happen automatically.

The Actions tab provides detailed progress for your workflow, so if something goes wrong it is a bit easier to debug than default GitHub Pages.

Just enjoy your newly built website!

Markdown in a Minute

Sat, 01 Aug 2020 00:00:00 +0000

You can start writing Markdown in one minute!

Markdown is a simple syntax for writing content for the web. The plain text code below demonstrates all the basics–click the button to view how it will look rendered into HTML. Check the resources at the bottom to learn more!

Toggle this Markdown:

# Heading One

## Heading Two

### Heading Three, etc.

Any text with no empty lines between will become a paragraph.
Leave an empty line between headings and paragraphs.

Font can be *Italic* or **Bold**.
Code can be highlighted with `backticks`.
Hyperlinks look like [GitHub Help](https://help.github.com/).

Images look similar:

![alt text here](https://upload.wikimedia.org/wikipedia/commons/4/4b/Focus_ubt.jpeg)

A bullet list is created using `*`, `+`, or `-`, like:

- dog
- cat
- muffin

A numbered list is created using a number + `.`, like:

1. one
2. two
6. three
2. four

> Block quote.
> Continuing the quote.

Horizontal rule:

-------

Heading One

Heading Two

Heading Three, etc.

Any text with no empty lines between will become a paragraph. Leave an empty line between headings and paragraphs.

Font can be Italic or Bold. Code can be highlighted with backticks. Hyperlinks look like GitHub Help.

Images look similar:

A bullet list is created using *, +, or -, like:

dog
cat
muffin

A numbered list is created using a number + ., like:

one
two
three
four

Block quote. Continuing the quote.

Horizontal rule:

Reference:

GitHub Mastering Markdown Guide
Markdown Tutorial
Markdown and Pandoc for Academic Writing
More _drafts notes

Online editors:

Dillinger
StackEdit
HackMD (real time collaboration)

Process Document with XML + XSL

Mon, 06 Jul 2020 00:00:00 +0000

People who work a lot with XML tend to use Oxygen XML Editor. Unfortunately, the XML ecosystem seems fairly stagnant outside of Oxygen. XSLT has version 1, 2, and 3 (draft)–but only Saxon has fully implemented versions 2 and 3.

I don’t work with XML a lot, so I don’t want to pay for Oxygen nor learn how to use it, but occasionally I have to process some XML using XSLT. So here are some options… (p.s. my favorite is to avoid the XSLT by parsing the XML with Ruby, then just outputting using Ruby instead)

xsltproc

For a quick and easy utility, install:

sudo apt install xsltproc

Then use the v.1 processor like:

xsltproc --output out.html stylesheet.xsl example.xml

Saxon

Saxon provides an open source and commercial version of their v.2 processor implemented in Java. If you install from a Linux repository, you will get a relatively old version–which in this case is actually good. Apparently more recent “open” versions from Saxonica have limited advanced features.

Be sure you have Java installed (default-jre), then install:

sudo apt install libsaxonb-java

Use is via a Java command like:

java net.sf.saxon.Transform -o:output.html -s:example.xml -xsl:test.xslt

Or via the commandline wrapper (man page):

saxonb-xslt -s:example.xml -xsl:stylesheet.xsl -o:output.html

If you run with a v1 stylesheet you may get a warning, but it still works fine. You will also get a warning for “unsafe extensions” for commands such as xsl:result-document. To ignore the warning, add -ext:on, like:

saxonb-xslt -s:example.xml -xsl:test.xsl -ext:on

Ruby Nokogiri

Nokogiri is a popular Ruby Gem for parsing HTML and XML documents. It implements v.1 features. It is fairly easy to set up a .rb file or Rake task to process batches of XML.

First, ensure you have a complete, up-to-date version of Ruby installed on your system. Then gem install nokogiri.

Using it looks something like:

require 'nokogiri'
document = Nokogiri::XML(File.read('example.xml'))
template = Nokogiri::XSLT(File.read('test.xsl'))
transformed_document = template.transform(document)
File.write('output.html', transformed_document)

i.e. open the XML; open the XSL; apply the XSLT; write the output to a file. See Nokogiri stylesheet docs.

Alternatively, rather than writing/applying an XSL, I have found it easier to just parse values out of the XML using selectors, then write the new document using Ruby instead.

In fact, I think I will do my best to avoid XML if possible…

pdfjam is Handy

Sun, 05 Jul 2020 00:00:00 +0000

pdfjam is a handy package for using the pdf utilities in LaTeX. I use it to quickly combine a batch of image files into a standard PDF. It can be installed as part of the TeX Live LaTeX distribution on Ubuntu:

sudo apt install texlive-extra-utils

(if you don’t have TeX Live installed, add texlive texlive-fonts-extra as well)

Once installed, you can check the full options: pdfjam --help

Now you can easily combine separate PDFs or images into a single PDF. The basic command is put together like pdfjam --options filenames page-selections.

For example, if I have a folder of JPG images I want to combine into a standard PDF:

pdfjam --letterpaper *.jpg

Let’s say I have three PDFs where I want to combine only some of the pages:

pdfjam --letterpaper example1.pdf 1-3 example2.pdf - example3.pdf 4-5

This will output a single PDF including pages 1 - 3 of example1, all pages of example2, and pages 4 - 5 of example3.

You can use any of these standard LaTeX paper sizes:

a0paper, a1paper, a2paper, a3paper, a4paper, a5paper, a6paper, b0paper, b1paper, b2paper, b3paper, b4paper, b5paper, b6paper, c0paper, c1paper, c2paper, c3paper, c4paper, c5paper, c6paper, b0j, b1j, b2j, b3j, b4j, b5j, b6j, ansiapaper, ansibpaper, ansicpaper, ansidpaper, ansiepaper, letterpaper, executivepaper, legalpaper

Or create a custom size using the option --papersize '{WIDTH,HEIGHT}'.

You can also add embedded metadata using options like --pdftitle STRING and --pdfauthor STRING.

Handy!

Note: in theory you can install it on Windows if you use MiKTeX via the MiKTeX console by searching the in the packages. However, it might be easier to get it to work using Ubuntu WSL instead.

Atom Editor Notes

Wed, 27 May 2020 00:00:00 +0000

Atom Editor project is no longer active, and was sunsetted by GitHub Dec 2022.

Some community led projects have forked the code, including Pulsar edit.

Atom is an open source code editor built by GitHub in the Electron framework. In the past, I used it for quite awhile before switching to VS Code, because I felt like Atom was too slow and clunky. However, in classic Microsoft style, VS Code is moving toward too much unnecessary/undesired functionality that gets in the way of the clean, quick, simple interface that attracted users to it in the first place. Now MS purchased GitHub, so the development status of Atom seems a bit unclear–does MS really want to develop both the leading Electron based text editors separately? Seeing the many unmaintained Packages makes me feel like Atom community is losing steam… In any case, I recently tried it again to see if I would like to switch back.

Install is easy and good on all OS. Click Edit > Preferences to start customizing.

Click on the Editor tab. I like to check “Show Indent Guide”, “Soft Wrap”, and “Soft Wrap Hanging Indent” 1. Spellcheck is built in which is handy.

Git / GitHub (builtin)

Atom comes bundled with GUI Git / GitHub tools which look similar to GitHub Desktop. Click the icons in the lower right or Ctrl+Shift+9 toggles the Git panel. Click on a file to view it’s diff.

Like GitHub Desktop an icon offers Fetch, but then changes to Push once you make some Commits, or Pull when there are remote updates. Right click the icon to access more options.

This seems like a real plus for people who might use GitHub Desktop instead. However, I prefer the efficiency of VS Code’s approach which adds version control to a tab in the Project browsing panel.

Markdown (builtin)

Atom has a Markdown preview builtin: just click Ctrl+Shift+M and a side-by-side live preview will appear. However, I like to tweak the settings to use GitHub’s standard Markdown styling. Go to Preferences > Packages, search for Markdown, click on markdown-preview, and check off “Use GitHub.com style”.

Terminal

Atom is missing a built in terminal, so you need to install one from the Community Packages. This seems like a major barrier to adoption since it frustrates many users. There is a confusingly large array of options, yet very few actively developed.

Open Edit > Preferences > +Install, and search for terminal to find options. In the past I used platformio-ide-terminal. However recently (as of Nov 2020) it no longer works for me on Ubuntu, nor do the many related terminals. platformio-ide-terminal adds an icon to the lower left and the key binding Ctrl+` to open the terminal.

The only terminal package that actually works for me anymore is x-terminal. x-terminal treats the terminal as a pane just like any other open editor–interesting approach, but I don’t think I like it. It uses the keybinding Ctrl+` to open a terminal, but doesn’t seem to add an icon anywhere.

Fira Code

I like to use Fira Code font in my editor. However, I do not like Fira Code’s fancy font ligatures–they just confuse me. Ligatures are enabled by default on Atom, so to turn them off you need to edit the config.

First, open Edit > Preferences > Editor, and add Fira Code to the front of the Font Family list. Then open Edit > Stylesheet, which will open a file styles.less in your config folder where you can add customizations. Inside the atom-text-editor selector, add the rule -webkit-font-feature-settings: "liga" off, "calt" off;. It will look like:

atom-text-editor {
  -webkit-font-feature-settings: "liga" off, "calt" off;
}

You should have Fira Code with no ligatures.

Handy keys

Use Ctrl + mouse wheel to zoom font size.

Windows Subsystem for Linux

Tue, 28 Apr 2020 00:00:00 +0000

Probably out of date? I haven’t used this for a long time!

On Windows I use Git Bash (from Git for Windows) for everyday commandline tasks. Occasionally I need something a bit more complete and have used Cygwin in the past. However, Windows Subsystem for Linux (WSL) is now offering a great alternative that is easy to install.

P.S. WSL 2 sounds even better, but I don’t have an up-to-date enough Windows built to try it. P.S.S. If you are using Bash commandline a lot, why not just try to linux fulltime or dual boot!

Install WSL

Check the official install docs for details, but recent improvements has made installing WSL relatively easy.

Enable WSL by opening PowerShell as admin and running Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux. You will probably have to restart.
Open “Microsoft Store” app and search for your preferred distro (check the author to ensure it is official). I suggest the latest LTS Ubuntu from “Canonical Group Limited”, currently 20.04 LTS.
On the app listing, click “Get” button then “Install”.
Once installed, click “Launch” on the app listing, or find and click in your Start Menu. This will open up a terminal window that says something about initializing. This first time startup will take awhile, a few minutes at least.
Once initialization finishes, a prompt will appear in the terminal to create a username and password. This will be your sudo admin user (and is not related to your Windows accounts).
Once your user is set, a normal terminal prompt will appear. Update your system since this is not done automatically (Ubuntu: sudo apt update, then sudo apt upgrade).
All ready! Now you can find it in your Start Menu, or from the context menu when holding Shift.

WSL Tips

From WSL terminal, your Windows files are at /mnt/c/.
The easiest way to get to a Windows file location is to hold Shift then right click in a folder in Explorer–the context menu will have the option “Open linux shell here”.
On the Windows side, your WSL files are somewhere in C:\Users\<username>\AppData\Local\Packages, but Windows says don’t mess with that. Up-to-date Window’s builds allow you to access WSL directly in Explorer at \\wsl$\. If you are in WSL terminal, type explore.exe . to open Explorer in that location.
Updates are not automatic, so update manually when you use it.
Uninstall is easy via Microsoft Store app.
If you forget your user password it is possible to reset via Powershell.

Ubuntu 20.04

Mon, 27 Apr 2020 00:00:00 +0000

New long term release Ubuntu 20.04 is out!

Long term releases get guaranteed five years of support, so are great if you like to install a solid system once and leave it. On the other hand, I like do a fresh install at every release, but either way, this note gathers up tips for getting everything set up after you install. I have a Dell XPS 15, Lenovo Flex, and HP Envy, and so far everything seems to work well–and the desktop seems noticeably faster than 18.04.

Install from USB

download Ubuntu 20.04
Burn ISO to a USB stick
Plug in your live USB into computer (while turned off)
Press power button and hit boot interrupt button (usually F12, Delete, Esc, or F2)
Look for boot options menu, then select your USB to boot from
After some thought, it should boot into the Ubuntu installer where you can choose to test or install. I usually test first, so you can play around with the live system to make sure it seems to be working, then click the installer from the live system.
During install, when it asks about hard drive, select the “Advanced options” to find LVM and encryption.

Settings

Open Settings:

Settings > Background to get your wallpaper set.
Settings > Appearance > checkout new “Dark” mode option and set Dock icon size.
Settings > File History & Trash > checkout new options for automatically deleting history, trash, and temp files.
Settings > Display > click the “Night Light” option at the top to configure a red shift in the evening.
Settings > Date & Time > use “Time Format” option to change 24hr or AM/PM.

Tweaks

Install Gnome Tweaks (GUI tool to customize desktop) and Gnome Extensions integration (enables using extensions via browser interface) right away, sudo apt install gnome-tweaks chrome-gnome-shell. (Note: gnome-tweaks used to be named gnome-tweak-tool)

Open Tweaks:

General > turn Animations off for faster performance.
Appearance > change themes.
Extensions > turn off Desktop icons.

Open Firefox browser:

Preferences > Privacy & Security > check “Use a primary password” and set. This ensures your saved passwords are encrypted on your device.
Add Firefox Multi-Account Containers, Facebook Container
Add adblocker such as uBlock Origin

Next you want some GNOME extensions. Earlier installations use GNOME extensions plugin and gnome-shell-integration on Firefox to install and manage extensions. However, in newer installations using the Firefox snap this option no longer works! Install gnome-shell-extension-manager instead which gives you an app to find, install, and manage extensions. Try Caffeine (disable screen lock to watch movies or just not have your laptop locking all the time) or check some other suggestions.

Enable firewall: sudo ufw enable

Next I like to have Gedit (“Text Editor”) customized so its ready as a quick editor. Open the app, then click the name in the top bar to open the app menu > Preferences. See customization suggestions notes.

Install Essentials

Check Ubuntu packages and install sudo apt install:

tlp tlp-rdw (advanced laptop power management)
development necessities: build-essential git curl

Note that some packages are transitioning to snaps instead traditional debs. Check snapcraft store for snap details or find them in Ubuntu Software. I like to look for the official “Canonical” + checkmark as developer, since you have to pay attention to who packaged the snaps. From snap install:

Chromium (browser)
Visual Studio Code or Atom (editor)
KeePassXC (password manager)

Install Languages

Note, 20.04 has some different system languages installed. It ships with Ruby 2.7 (new) and Python 3.8 (no more Python 2). In the terminal, Python is python3, so which python will come up with nothing.

Ruby: rbenv or RVM set up
Python: Anaconda set up
Java: sudo apt install default-jre or default-jdk (this will be OpenJDK 11+, fine for apps like OpenRefine, if you need more specific versions search packages)
Node: use nvm (this seems to be the best method; install nvm, then nvm install --lts), or official NodeSource ppa, or repository version nodejs (which is out of date)

CONTENTdm metadata export tips

Sat, 04 Apr 2020 00:00:00 +0000

Many libraries use CONTENTdm (CDM) to create and host their digital collections. Because many workflows include creating and updating metadata in project client, your CDM repository may contain metadata that is not reflected in other sources. Additionally, you may want to get CDM numbers out of the system for your objects so that you can use the CDM API to access them–or to create “skins” like CollectionBuilder-CONTENTdm.

The most efficient way to get this metadata is to export it from the CDM admin interface. These tips assume you have access to a CDM instance’s admin interface–if you do not have access, check CDM api tips for other ways to harvest metadata.

Export collection metadata

Log into your CDM admin (at a url something like “https://server12345.contentdm.oclc.org/cgi-bin/admin/start.exe” using your associated OCLC account).
Click on “Collections” tab.
Select desired collection from the drop down menu on left, then click “Change” to switch to that collection.
Click on “export” in the nav under the “Collections” tab.
On Export metadata page, choose “Tab-delimited”, check “Return field names in first record” option, and then click “next”.
This will lead to an “Export successful” page. Right click on the “export.txt” and select “Save link as” from your context menu. This should open a file save dialog box on your computer–I usually rename the file from “export.txt” to something meaningful including the collection name and export date (“example_2020-12.txt”).

At this point I also like to record some collection information for reference:

Click “reports”, then “Item Types” to see total number of items and file types.You will want this information to check the accuracy of your metadata once you start processing the export.
Click “profile” to find the “Collection alias” (e.g. /example). This will be necessary to construct API calls.
Click “fields” to review the metadata template for the collection so you know what should be in the export.

Reformat “export.txt”

“Tab-delimited” text (also commonly called “tab-separated values” or TSV) is a plain text tabular data format, using tabs as the delimiter between columns. Like “Comma-separated values” (CSV), it is a convenient format for data exchange often used to store spreadsheet data, but is often implemented without a formal spec.

The tab-delimited “export.txt” from CDM uses UTF-8 encoding and LF line endings, which are the standard on the web and Linux, but not well supported in default Windows applications. If you open the file directly with Excel, it has a good chance of being corrupted (messing up special characters or line breaks). Collections that include a full text field (e.g. extracted from PDFs) are particularly challenging to get parsed correctly. These errors are often hard to detect, so it is important to take care in initial handling the “export.txt” and reformat to a more useable file type.

Here is some suggestion workflows:

OpenRefine (free and open):

Start OpenRefine.
Select your “export.txt” and click “Next”.
On the “Configure Parsing Options”, the preview will likely be all wrong. Choose all these options to parse correctly:
- In “Parse data as” select “CSV / TSV / separator-based files”.
- In “Character encoding” click the box and select “UTF-8” (this is essential to avoid losing special characters on Windows).
- In “Columns are separated by” select “tabs (TSV)”.
- Uncheck the option ‘Use character “ to enclose cells containing column separators’ (this is essential to avoid scrambled or losing records!)
- Click “Create Project” to finish parsing / importing the data
Once set up, use Refine’s “Export” button to cleanly reformat. If you would like to work with Excel, use “.xlsx” option. For preservation and interchange, “Comma-separated value” is best option.

LibreOffice Calc (free and open):

Open LibreOffice Calc, then File > Open and choose your “export.txt”
This will open the “Text Import” parsing options, choose these options:
- “Character set” select “Unicode (UTF-8)”
- “Separator Options” choose “Separated by” and check only “Tab”.
- “Fields” use the box to select all the columns, then select “Column type” “text” (this will prevent it from reformatting dates and numbers).
- Click “Okay” to finish parsing.
Calc will allow you to safely save as CSV, ODS, or XLSX.

Google Sheets:

Open a new blank Sheet.
Click “File” > “Import” > “Upload”, and choose your “export.txt”.
This will open the “Import file”, choose these options:
- “Import location” select “Replace current sheet”.
- “Separator type” select “Tab”.
- “Convert text to numbers, dates, and formulas” select “No” (to avoid reformatting dates and numbers and running into issues with full text field).

Excel (different versions have the menu options labeled differently, so these steps are just a vague approximation. As far as I can tell online Excel 365 can’t import data and can’t really be used for this process at all.):

Open a new blank spreadsheet in Excel (do not double click “export.txt” or open with Excel directly)
Click “Data” > “From Text/CSV” (in “Get data” area left side of ribbon)
This should pop open an “Import” options menu similar to the one described above for LibreOffice Calc–choose similar options. In “File origin” be sure to choose “Unicode (UTF-8)” or the parsing will have errors. (Excel 365 flashes some notices like “connecting” and “loading” data, as if you are setting up a live database connection–you can ignore this mysticism, its just importing data like normal).

Metadata notes

Admin fields: Exports contain a set of administrative metadata fields added by CDM. These are the last columns of the export, including: “Filename,OCLC number,Date created,Date modified,Reference URL,CONTENTdm number,CONTENTdm file name,CONTENTdm file path”. “CONTENTdm number” is the most useful, since this provides the “pointer” used along with the collection “alias” in API calls. The “Full Text” field is also usually auto generated by extracting text from uploaded items.

Compound objects: If your collection contains Compound objects (image or pdf), there will be a row for the object description, as well as each page of the compound object. The main object descriptions can be identified by checking “CONTENTdm file name” column for the extension .cpd. Page level item rows follow the main item, but can be difficult to disambiguate from normal items. Usually page level items do not have the full metadata filled out, so use a column always filled in object descriptions but blank in page level to isolate them. If you have extensive page level metadata this can be complex.

CDM Terms

CDM uses some terminology for the components of their API:

CDM server number: each hosted CDM has a number which is used in two different URLs in the “contentdm.oclc.org” subdomain, which are used for different types of API calls.
- “CDM server URL” - this is where your CDM admin is hosted, named “server” + server number, e.g. https://server12345.contentdm.oclc.org/. In general collection level API calls have to go to this URL.
- “CDM website URL” - this is where your public CDM instance is exposed, named “cdm” + server number, e.g. https://cdm12345.contentdm.oclc.org/. Many organizations have a custom domain in front of this subdomain which should work for most API calls. In general item level API calls have to go to this URL.
“Alias” - the identifier name for an individual collection, used in URLs. If set by CDM they look like p17254coll3, or can be manually created looking like argonaut.
“pointer” / “CONTENTdm number” - the identifier number of an individual item or object assigned by the CDM database, e.g. 14. If harvesting metadata, this value is usually called “pointer” in XML. However, it is found in metadata export as “CONTENTdm number”.

If you look at the URL for at a CDM item page you can find the information. For example, in:

https://cdm17254.contentdm.oclc.org/digital/collection/p17254coll3/id/155/rec/1

The “CDM website URL” is https://cdm17254.contentdm.oclc.org, collection “Alias” is p17254coll3, and “pointer” is 155.

Check CDM API tips for API recipes to use / harvest digital objects.

Tea Data

Mon, 09 Mar 2020 00:00:00 +0000

This study seeks to find the best flavor black tea at the best value price point for bulk drinking at work in the library.

Methodology

Two library employees with extensive experience in tea drinking (i.e. drink at least one pot of tea per day) took part in a tasting session featuring seven widely available varieties of black tea. Each tea was prepared according to the manufacturer instructions printed on the box. A standard measure of milk was added to each prepared tea after the brew time was complete.

Each tester sampled the tea taking several moments to savor the flavors. Testers general impressions were discussed and recorded, as well as an overall rating out of five possible points. The highest rating awarded by individual testers was four (Red Rose and Taylors Yorkshire), and the lowest was one (Tazo Awake).

Prices of tea brands vary significantly at each grocery chain. To find a standardized price for comparisons, we chose to use the price at Walmart which carried all brands tested.

Best Value

In attempt to evaluate the best overall value, we used the equation:

Mean Rating / Price per Tea Bag

Base Value Ratings

Tea	Value Rating
Red Rose	1.077586207
Taylors Yorkshire Red Tea	0.7155635063
Tetley British Blend	0.6711409396
PG Xtra Strong	0.6276150628
Twinnings Irish Breakfast	0.5573248408
Twinnings English Breakfast	0.398089172
Tazo Awake English Breakfast Tea	0.2090301003

This surfaced two surprising results:

Best value was the rather unassuming Red Rose Tea, which tastes pretty good yet is available very cheap.
Worst value was the popular Tazo Awake, very often served at conferences and hotels, which tastes terrible yet is very expensive.

Complete Tea Data

table powered by simple-datatables

Detect Website Platform

Mon, 03 Feb 2020 00:00:00 +0000

In January 2020 I completed a quick scan of university library websites to include in a paper I was writing. The data and method are shared as a repository, library_websites_scan, or on Zenodo doi:10.5281/zenodo.3653184.

I wanted to share part of the method here, since it is basically just tips on how to inspect a web page and determine the platform used to build it.

Platform signatures

To determine the platform, use “View Page Source” and “Inspect” developer tools to look at the base HTML code. Check the HTML for platform signatures. Occasionally, the main index page has more customization and added features than other content pages, thus is harder to determine the platform–in that case, visit a less busy page to view the code.

Drupal based sites:

usually include the generator meta tag with the major version number,
- <meta name="Generator" content="Drupal 8 (https://www.drupal.org)" />
- <meta name="generator" content="Drupal 7 (https://www.drupal.org)" />
older sites usually load CSS as a huge series of import statements, with the file paths including “/modules/”, e.g. @import url("https://example.edu/modules/system/system.base.css?q4zjuk");
newer sites usually load CSS as a smaller series of CSS files, with the file paths including “/files/”, and filenames with a bunch of random characters, e.g. ` `
usually have a JS file, JS object, and element attributes that include “drupal”, e.g. data-drupal-selector="drupal-settings-json"
often use links/src that contain “/node/”

WordPress based sites:

usually include the generator meta tag with version number,
- <meta name="generator" content="WordPress 4.9.10" />
- <meta name="generator" content="WordPress 5.3.2" />
usually contain the short link “s.w.org” (which resolves to wordpress.org), e.g. <link rel='dns-prefetch' href='//s.w.org' />
usually use asset paths that contain “wp-“, e.g. href='//example.edu/wp-content/themes/eu-2014/style.css?ver=3.6'

ASP.NET based sites:

usually include classes that include aspNet, e.g. class="aspNetHidden".
usually contain “viewstate” variables such as <input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="0DhWp/KtK1Wf-some-crazy-string" /> and other ids with “__” in front of the name.
usually contain links/src ending in .aspx or .axd
may contain a generator meta tag giving a specific CMS platform, e.g. <meta name="Generator" content="Sitefinity 11.2.6900.0 PE" />
may use asset paths that include a specific CMS name, e.g. src="https://example.azureedge.net/sitefinity-production/images/example-icon.png?sfvrsn=11106867"

PHP based sites:

contain links/src ending in .php

Quick Local Static Servers

Wed, 29 Jan 2020 00:00:00 +0000

Three quick local static web server for dev or what ever.

Python

Use http.server from the standard library:

python -m http.server 8000 then surf to localhost:8000

(in python 2: python -m SimpleHTTPServer)

Ruby

Use Ruby WEBrick server with handy command from standard “un” library:

ruby -run -e httpd . -p 8080 then surf to localhost:8080

(see Run, Ruby, Run for details of this idiosyncratic command)

Node

Use http-server package. Install npm install -g http-server. Then:

http-server in the directory to serve

Ruby RVM Notes

Sat, 04 Jan 2020 00:00:00 +0000

Ruby is an interesting language, but can be a bit frustrating with the many versions and incompatibilities. My main use is for working with Jekyll or Ruby on Rails.

On Windows I always use RubyInstaller, which seems to work well.

On Linux, I have tried the version in the distro repositories (Ubuntu: ruby-full), but this is usually fairly out-of-date and can cause some headaches (such as requiring sudo to install gems, unless you edit your bash profile, see Jekyll Ubuntu install docs). On Ubuntu 19.10 ruby-full will be version 2.5.1, and updates are rare.

A better, easy alternative is to use the official Ruby snap package maintained by the Ruby core team which stays up-to-date. Install using sudo snap install ruby –classic (note: the -classic flag means the snap will not be “confined”, since Ruby will need full access to your system to function. Snaps requiring -classic are audited by the repository). The snap will keep itself up-to-date, so may change versions on you. You may need to install additional build tools as well (at least ` build-essential zlib1g-dev`).

However, you might want to use a Ruby version manager to keep your dependencies in order and separate from your system Ruby. I have used RVM described below–however, more recently switched to using rbenv (see rbenv notes).

When installing RVM on Ubuntu there can be some barriers because the way RVM functions isn’t compatible with the default set up of Gnome terminal–either you can reconfigure, or do a little workaround by setting up a second terminal.

Set up RVM on Ubuntu

Install curl and build-essential if you don’t have them (sudo apt install curl build-essential).

Set up Tilix:

Install Tilix terminal emulator (sudo apt install tilix)
Open Tilix
Click on the user name above the terminal window, select Profiles > Edit Profile.
Click on Command tab, check “Run command as a login shell”
Close Tilix

Open Tilix and install RVM using the official instructions (note: there is a PPA for Ubuntu that can take care of everything).

Install a Ruby version, probably the most recent stable (listed on Ruby downloads), for example: rvm install 2.7.0. This will download all the dependencies and build the Ruby version, which might take awhile…

When it completes, check your Ruby, ruby -v. If this is your first version, it will become the default, or you can select one, rvm use 2.7.0. If you want to generate the documentation, run rvm docs generate-ri.

Now you can install Jekyll: gem install bundler jekyll

Once you restart your system, Gnome terminal will recognize your current Ruby version. Use RVM from Tilix.

Intros:

Introduction to CDIL

Mon, 11 Nov 2019 00:00:00 +0000

Notes and links for Honors visit to CDIL. The Center for Digital Inquiry and Learning (CDIL) is a collaboration between the College of Letters, Arts and Social Sciences (CLASS) and the University of Idaho Library. Its purpose is to advance digital scholarship opportunities at the University of Idaho.

Introductions

Introductions
Digital Humanities and Digital Scholarship???
What is DH
Center for Digital Inquiry and Learning (CDIL)

CDIL Projects

Learning, outreach, events:

Research, publishing, digital collections:

Historical Japanese Ceramic Comparative Collection (Graduate fellow Renae Campbell, Historical Archaeology)
Closure of Syringa (Undergraduate fellow Denessy Rodriguez, Sociology & Anthropology)
CTRL+SHIFT (Devin Becker, graduate fellows Corey Oglesby and Lauren Westerfield, MFA)
CCC in Idaho (Patricia Hart, JAMM)

Software:

Virtual Reality

Oculus Rift
HTC Vive
Sony PlayStation VR

Intro to Raspberry Pi

Mon, 07 Oct 2019 00:00:00 +0000

Please see newer version of this outline. Some content below is out of date.

Arduino vs. Raspberry Pi

Both are very flexible, with great documentation and community.

Arduino:

microcontroller, 8-bit, tiny memory
very simple, solid, reliable
very low power consumption
do one thing really well

Raspberry Pi:

Single board computer, 32/64-bit CPU, 512MB - 4GB RAM
Connect: USB, Ethernet, Wifi, Bluetooth
Output: Video (HDMI), audio (HDMI, 3.5mm jack), GPIO pins
Full operating system
Use high-level programming languages (mainly Python)
can do MUCH more than Arduino, but…
- consumes much more power and requires steady power supply
- OS requires updates
- less reliable overall

e.g. poemBot

Raspberry Pi starter kit

Boards (A, B, Zero, Compute, spec chart)
Power supply (you need a good one! A rainbow icon shows up on screen if power goes low)
SD card (try to get decent quality, high read/write speed, “SDHC, Class 10 / UHS 3”, min 8GB)
Console cable
Keyboard, mouse, HDMI cord
add ons / HATs
ribbon cable + breadboard

Get Started

First step is to choose an OS for your Pi and burn it to your SD card. See official step-by-step for a visual walk through. Most likely you will want to start with NOOBS (does not require burning image) or an image of Raspbian:

NOOBS

NOOBS is an automatic Pi OS installer designed to be super simple
Download NOOBS, unzip, copy to SD card (you do NOT burn an image, just copy)
Plugin SD, connect video, keyboard, and power to Pi
Follow instructions!

Burn image to SD

See Burn ISO note
Download Raspbian (headless or desktop) (Linux distro for Pi, based on Debian, see Intro to Linux note)
SD Formatter
Etcher
Plug SD into Pi, boot up!
see Raspbian docs

Use Pi

GUI (i.e. plugin HDMI, this is often easiest way to get started, even if you are planning to go headless)
console cable (can be handy for developing headless projects, can provide power to the board)
SSH (not possible on university wifi) ssh [email protected]
- enable on Pi
- find IP address
- use SSH
- copy files to Pi: scp filename [email protected]:~/destination/foldername/
- copy files from Pi: scp [email protected]:~/source/file /destination/directory/
VNC (virtual desktop)
- enable/install on Pi
- install client, use cloud service
Tip: plug SD into computer to directly edit file system!
/etc/rc.local file runs at boot, so you can add command line calls to start your program

Resources

Offical Raspberry Pi Documentation, Help, and Forum
Adafruit Learn, pi category
Play around to learn more: Volumio music center, RetroPi

Visual Studio Code Notes

Tue, 01 Oct 2019 00:00:00 +0000

Code is an open source text editor from microsoft (docs). It is (was??) light weight compared to most traditional IDE, yet fully featured and efficient. I find it perfect for working on web projects (i.e. Open Folder not a file), such as using Jekyll, with Git nicely integrated.

The VSCodium project provides an alternative all open source + no tracking version of VS Code. VS Code is open source, however, the installers you download from MS include additional non-open-source components and extra MS tracking features.

Update: unfortunately MS seems to be continuously adding more and more features (i.e. junk) all the time–taking it from a nice minimalist editor, to yet another bloated piece of software. I am still using it, but starting to get frustrated by more and more features that I don’t want and actually get in the way of working. Almost every update I have to figure out how to turn off another feature. Typical MS…

To configure the editor click the cog icon in the lower left and choose Settings (or go to File > Preferences > Settings, or Ctrl + ,). This will bring up the visual settings editor which can be searched to find options. By clicking the file icon in the upper right of the Settings pane will switch to the settings.json view. This allows you to view the key + value pairs representing your settings, or allow you to paste in new ones that will override the default settings. This is very handy for sharing your set up–just copy the values and save in a text file, then paste the values on a different computer (see my version pasted at the bottom). The Settings are cross-platform, so they will work on any computer, with a few that are specific to each OS.

Here are my notes-to-self about stuff:

Config preferences

A few highlights:

Enable word wrap: "editor.wordWrap": "on"
Get rid of excessive stuff in file explorer: "problems.decorations.enabled": false
Hide minimap, sometimes I find the map annoyingly in the way: "editor.minimap.enabled": false
Disable clearing clipboard if you accidentally Ctrl+C on blank line: "editor.emptySelectionClipboard": false
Disable annoying drag and drop selection moving: "editor.dragAndDrop": false
Disable annoyingly unhelpful html tag closing: "html.autoClosingTags": false
Disable annoying persistent terminal sessions which cause endless errors: "terminal.integrated.enablePersistentSessions": false
Disable weird git visualization that is super distracting in source control panel: "scm.showHistoryGraph": false

There is an ever increasing number of “helpful” “smart” auto select and delete things (seemingly modelled after the utterly infuriating and worthless features of MS Word) regularly added, so I end up spending a lot of time trying to turn things off. But, once you have everything set up nicely, copy your settings.json file to a git repo and reuse it on all your computers! (example at the bottom of this note)

Integrated terminal

Access a terminal without leaving the editor by typing Ctrl + ` (control + backtick). This is extremely handy when developing Jekyll projects, since you can jekyll s and have the output easily visible while you are editing. I also use it for any non-routine Git activity, since the built in GUI client is only good for basic git add/commit/push equivalents.

The terminal is very straight forward (automatic) on Linux or Mac, but on Windows the default terminal is horrible old cmd. The first time you open the terminal on Windows, it should prompt you to with options to change to other terminals you have installed (i.e. Git Bash, WSL Bash, or PowerShell). If you don’t get the prompt, click the drop down on the left side of the terminal window that says “Command Prompt” and select “Select Default Shell”. This should open a list of option in the “command pallet” at the top of the window. Or override the setting "terminal.integrated.shell.windows": "C:\\Windows\\system32\\cmd.exe" with the path to the terminal you want to use.

Extensions

Code has a growing ecosystem of extensions. Click the Extensions icon on the left side and search in the box to find new ones.

Spell check: I use “Code Spell Checker”, and it seems really good and avoids a lot of code-related false positives.
CSV highlighter: “Rainbow CSV”, this is incredibly helpful for looking at CSVs.
Data viz: SandDance for VSCode (handy for looking at CSV contents)
Arduino: If you have Arduino IDE installed on your system, you can add similar functionality to Code. Search for “Arduino” from Microsoft. Now when you open a .ino sketch, you will have options at the bottom of the window to select your board and port. More info vscode-arduino.
XML: “XML Language Support”

Multi-cursor

To add more cursors to work on multiple lines at once, Alt and left-click (or toggle to Ctrl + left-click in Selection menu).

HTML

Auto format html, highlight area and type Ctrl+K followed by Ctrl+F (or maybe Ctrl+Shift+I). Setting for auto format are in Settings, add "html.format.wrapLineLength": 0, to prevent auto link breaks.

Emmet snippet expansion: write the abbreviation and press tab (Emmet Cheatsheet).

Markdown preview

Click Ctrl+Shift+V to preview rendered file.

Change themes

Type Ctrl+K then Ctrl+T, then choose a theme from the list. Get more themes from the Extensions shop.

Settings

{
    "editor.fontFamily": "'Fira Code', 'Droid Sans Mono', 'monospace', monospace, 'Droid Sans Fallback'",
    "editor.wordWrap": "on",
    "breadcrumbs.enabled": false,
    "editor.autoClosingBrackets": "never",
    "editor.autoClosingQuotes": "never",
    "editor.autoSurround": "never",
    "editor.codeLens": false,
    "editor.colorDecorators": false,
    "editor.copyWithSyntaxHighlighting": false,
    "editor.dragAndDrop": false,
    "editor.emptySelectionClipboard": false,
    "editor.links": false,
    "editor.acceptSuggestionOnCommitCharacter": false,
    "editor.acceptSuggestionOnEnter": "off",
    "editor.quickSuggestionsDelay": 100,
    "workbench.enableExperiments": false,
    "workbench.startupEditor": "none",
    "workbench.statusBar.feedback.visible": false,
    "explorer.enableDragAndDrop": false,
    "outline.problems.badges": false,
    "outline.problems.enabled": false,
    "outline.problems.colors": false,
    "extensions.ignoreRecommendations": true,
    "problems.autoReveal": false,
    "html.autoClosingTags": false,
    "html.format.maxPreserveNewLines": 2,
    "terminal.integrated.cursorBlinking": true,
    "terminal.integrated.rightClickBehavior": "default",
    "html.format.wrapLineLength": 0,
    "git.terminalAuthentication": false,
    "cSpell.enabledLanguageIds": [
        "asciidoc",
        "c",
        "cpp",
        "csharp",
        "css",
        "git-commit",
        "go",
        "handlebars",
        "haskell",
        "html",
        "jade",
        "java",
        "javascript",
        "javascriptreact",
        "json",
        "jsonc",
        "latex",
        "less",
        "markdown",
        "php",
        "plaintext",
        "python",
        "pug",
        "restructuredtext",
        "rust",
        "scala",
        "scss",
        "text",
        "typescript",
        "typescriptreact",
        "yaml",
        "yml",
        "xml"
    ],
    //"terminal.integrated.sendKeybindingsToShell": true,
    "problems.decorations.enabled": false,
    "rainbow_csv.enable_tooltip": false,
    "rainbow_csv.enable_tooltip_warnings": false,
    "rainbow_csv.enable_tooltip_column_names": false,
    "terminal.integrated.enableFileLinks": false,
    "terminal.integrated.experimentalLinkProvider": false,
    "terminal.integrated.altClickMovesCursor": false,
    "terminal.integrated.gpuAcceleration": "off",
    "workbench.settings.enableNaturalLanguageSearch": false,
    "telemetry.telemetryLevel": "off",
    "cSpell.enableFiletypes": [
        "csv"
    ],
    "html.autoCreateQuotes": false,
    "editor.autoClosingDelete": "never",
    "editor.autoClosingOvertype": "never",
    "workbench.editor.enablePreview": false,
    "terminal.integrated.enablePersistentSessions": false,
    "terminal.integrated.persistentSessionReviveProcess": "never",
    "editor.smartSelect.selectLeadingAndTrailingWhitespace": false,
    "editor.smartSelect.selectSubwords": false,
    "editor.minimap.enabled": false,
    "editor.unicodeHighlight.ambiguousCharacters": false,
    "editor.pasteAs.showPasteSelector": "never",
    "markdown.editor.filePaste.enabled": "never",
    "markdown.editor.pasteUrlAsFormattedLink.enabled": "never",
    "scm.showHistoryGraph": false
}

Old Notes

A few things are easier now–below are some older notes about configuring things the hard way.

To use Git Bash as Windows terminal, look in Git’s bin directory in the program files (on Win 10 this should be C:\Program Files\Git\bin or something more complicated on Win 7 like C:\Users\username\AppData\Local\Programs\Git\bin). In your settings replace the cmd path with the full path to bash.exe, and escape the back slashes, e.g. "terminal.integrated.shell.windows": "C:\\Program Files\\Git\\bin\\bash.exe".

(note: shortcuts to Git Bash terminal on your desktop and start menu will point to C:\Program Files\Git\git-bash.exe, the MinTTY terminal. If you use that app, Code will open a new external terminal. To use Git Bash on the built in terminal make sure you are pointing to the bin\bash.exe).

Library Intro Notes

Wed, 11 Sep 2019 00:00:00 +0000

Notes for graduate student meeting introducing library resources.

Library Services

Get help:

Reference Desk (Ask a reference librarian: walk up, chat, email, phone, don’t be shy! Hours)
Library website! (there is a ton of information and resources here)

Resources:

Summit (network of 37 NW academic libraries in Orbis Cascade Alliance) and ILL (worldwide network for library sharing)
Reserves (save your students money!)
Book purchase requests

More stuff:

Library Catalog

Primo Discovery

Content: everything that the library has access to (books, journals, gov docs, articles, physical and digital)
Less focused and powerful than databases
Use notes:
- sign in (access resources off campus)
- account (check loan status, saved items/searches/history)
- scopes (UI Library, Summit, Articles, Reserves)
- filters (date, resource type, etc)
- “personalize” (weights search results in your discipline)
- item features (citation, permalink, email, summit libraries, virtual browse)
- advanced search, personalize, facets

Databases

Database A-Z List (all databases that the Library subscribes to)

Content: each database has a specialized set of materials with enhanced metadata to aid in discovery.
Use notes:
- databases may not include full text access
- each may have different limitations and restrictions
- use proxy to access from off campus

Important ENGR databases:

IEEE Xplore (standards, personalized account)
Engineering Village / Compendex
ACM Digital Library (conference proceedings)
Dissertations & Theses PQDT (ETD database for most universities)
Safari: O’Reilly Learning Platform (handy tech learning resources)
SciFinder (patents, chemical abstracts)

Research Concepts

Scholarly literature:

Peer Review
Grey Lit
Preprint archives, e.g. arXiv or engrXiv

Citation:

Citation guide
Citation management software (tools to store citations and create bibliographies. Start using one now, it will make your life so much easier!)

General search strategies:

Booleans (AND, OR, NOT)
Keywords and concept mapping (broader, narrower, related, synonyms / thesaurus. intro video)
Citation chaining (look at what an important paper in your area cites and who cites it)

Web of Science

Content: wide coverage of science scholarly/peer-reviewed articles from the most important journals and conference proceedings
- Index only (metadata, not full text), contains items that the Library might not have access to.
Excellent citation tracking, enables in depth citation chaining
Filter by highly cited, “hot in the field”, etc., to explore subject area
Export data, citation lists
WoS Search intro video, quick ref

Google Scholar

Content: good content coverage of academic literature, but not all peer-reviewed and sources are not always clearly labeled
- Index only (metadata, not full text), contains items that the Library might not have access to.
Familiar / easy to search
Good citation tracking
Connect to UI Library:
- Adding UI Library to your Scholar account will provide a link to full text content in search results via our subscriptions.
- Log in to Google Scholar with your Gmail account; From menu in upper left, go to settings > Library links; Search for University of Idaho, and click the check box; Click “Save”.

Handy web dev tools

Mon, 29 Jul 2019 00:00:00 +0000

Key shortcuts:

Paste plain text (get rid of weird formatting): Ctrl + Shift + V
Override a page’s custom context menu (so you can access your normal one): Shift + right click

Browser support:

Tests:

Web Vitals
Lighthouse tests
Page Speed
mobile-friendly
Rich Results Test (replaces discontinued Structured Data Testing Tool)
Facebook Sharing Debugger
webpagetest (test on actual devices and connections)

Web Master utilities:

Colors:

Accessibility:

Who Can Use (color checker)
Accessibility Developer Guide
ARL web accessibility toolkit
Usability.gov
Chrome Accessibility Developer Tools
WAVE tool
W3C Easy Checks documentation
Accessibility testing and tools (MDN web docs)
captioning videos for free

Stop using “click here”:

Also, don’t use link shorteners

Free images:

Image optimize:

For one off, Page Speed has a little link at the bottom of the analysis to download optimized images, css, and js (see Optimize Image)
Essential Image Optimization by Addy Osmani
Squoosh web app
Image Magick (notes)

Reference:

JQuery?

Design:

Quick local server:

Python 3: python -m http.server 8000 then surf to localhost:8000 (part of python standard library)
Ruby: ruby -run -e httpd . -p 8080 then surf to localhost:8080 (note in ruby 3.0+ webrick is removed from the standard library, so you will have to install webrick gem)
Node: install npm install -g http-server, run http-server

Ultra minimal computing notebook:

new-note (see article)
or paste into address bar data:text/html, <html contenteditable style='padding:10%;'>, type, then save with Ctrl+S.

Minimal hosting:

sdnotes

Find good font for your editor:

Programming Fonts, test out mono fonts in web based editor

Check content of certificate file:

openssl x509 -in libcal_uidaho_edu_cert.cer -noout -text

Git Intro

Thu, 27 Jun 2019 00:00:00 +0000

Introduce using Git and GitHub for collaborating (workshop outline)

Finish Prep (git + github account)
Take Survey

Why use Git and GitHub?

Git is a free, distributed version control system originally developed for coordinating huge software development projects (i.e. the Linux Kernel led by Linux Foundation).

However, Git paired with GitHub is also great for academic and personal uses, such as collaborating over code for your research project, drafting articles, or creating lab websites.

Intro to version control:

Git for science:

Reproducibility: records all steps and participants, ability to share complete history.
Backup: it is very difficult to delete history–if you commit it, Git stores it with integrity.
Collaborate: manage complicated work in parallel with multiple people/computers.
Experiment: branching makes you feel brave and free to explore!

GitHub:

Project management features
Organizations
Community (find code, connect)
Sharing (DOIs for citable code / data)

Basic Workflow

Git is best learned hands-on. Intro is via command line to give a clear step-by-step overview of how it works–in the future you may use a GUI tool or functions built into your IDE.

Keep in your heart that Git keeps track of Everything! Have peace of mind that you can’t loose your history once committed to a Git repo, unless you try really hard.

Be sure to .gitignore cache files and temporary outputs–just commit the source code, data, and notes.

Basic GitHub Collaborating Workflows

There is two basic workflows to collaborate on a GitHub repository:

Add collaborators to repo (simple, typical of smaller projects)
- On GitHub, click the “Settings” tab of your repository.
- On the left menu, click “Collaborators”.
- Add collaborators via email or GitHub name.
- Collaborator will need to accept the invite.
- Clone the repo to your local machine.
- Now you all have equal control over the repo content: push, pull, merge, etc.
- Using feature branches may be helpful to organize your work–create a branch, do some work, push the new branch to GitHub, then create a Pull Request to discuss with your team.
Fork and Pull Request (more complex, centralized control, typical of bigger projects)
- Navigate to your partner’s repo on GitHub.
- Click “Fork” in upper right.
- Make changes in your personal fork of the repo.
- On your personal fork, click “New pull request” button.
- Check the changes, click “Create pull request” button.
- Create a message saying exactly what changes you made and why.
- The original repo will now have a PR that collaborators can view and comment on. Only the owner can accept the request and merge it.

See GitHub Help Fork a Repo, About Pull requests, and Understanding the GitHub Flow for more info. Also, check out Atlassian’s Comparing Workflows or GitHub About collaborative development models for more options.

Collaborating Practice

Add collaborators:

create a new repository with README
clone to your local machine
add partner as collaborator
accept partner’s invite (check notifications)
clone partner’s repo
make change locally to partner’s repo README and push

Auto merge:

make change locally in your repo, creating a new file test.txt
try to push
pull (auto merge message)
push

Conflicts:

pull your partner’s repo
make a change to test.txt locally in your partner’s repo
push
make a change to test.txt locally in your repo
try to push
pull, enter merging state
fix conflicts
commit
push

Art of the commit: it is best to create small, targeted commits when collaborating. Each commit should do one specific thing making it easier for others to understand your work and navigate the history if necessary.

RStudio Integration

RStudio has builtin integration with Git, allowing you do complete the basic commands with the interface. Basically, an RStudio “Project” will equal a Git repo. You can directly clone a repo from GitHub using File > New Project > Version Control > Git. When you open a Project that is a git repo, you should see the “Git” menu on the interface giving access to the git commands.

RStudio support, Version Control with Git and SVN
SWC, Using Git from RStudio
Happy Git and GitHub for the useR

Project management features

GitHub adds many handy web-based features to manage your projects / Git repositories:

Issues. Create an issue to discuss and track ideas, bugs, projects, requests, etc. Can be assigned to people, tagged, and more. Also allows people outside of the project report problems with your code. Be sure to create checklists in the first comment box (- [ ] step)–they become click-able and show progress in the Issue view.
Projects. Create Trello board like lists to organize work.
Wiki. Simple wiki-style documentation that can be edited by your collaborators (note: written in Markdown not wikitext).

To make the most of these features, you will want to learn Markdown because it’s a great way to write and is used everywhere on GitHub. Issues, PR, and commits can be mentioned in any GitHub comment and will be replaced by reference shortlinks. GitHub users can be mentioned using @ and will be notified of your comment.

gh-pages

GitHub also offers free web hosting for your project, organization, or personal profile. Check workshop Go-go gh-pages!

Also, use GitHub Gist to instantly share simple notes, outlines, snippets, etc.

Citable Code

GitHub repos can be integrated with Zenodo to issue a DOI. DOI are a persistent identifier used in academic writing to cite articles and other works. Having a DOI for your code makes it easier to track citations and impact. Learn how at Making Your Code Citable.

License

Choosing a license is important if you want share data, code, and content. It ensures users have legal rights to reuse, and reserves your rights as the creator. The FAIR Principles point out that having a clear license is a requirement for legal interoperability of data.

It is convention to add a file as LICENSE or LICENSE.md to a repository so that people can easily find the license. GitHub can automatically generate some open source licenses for you, see Adding a license to a repository.

Choose an open source license (for software)
Creative Commons Choose a License (more common for text and data)

Resources

Software Carpentry, Version Control with Git
Data Carpentry for discipline specific data lessons
Git Book
Git for Teams
Safari Books Online

Random Interesting Fonts

Wed, 29 May 2019 00:00:00 +0000

Programming Fonts, test out mono fonts in web based editor
Public Sans, open font created by US government, on available on GitHub, “A strong, neutral typeface for interfaces, text, and headings.”
Inter, open font “specially designed for user interfaces with focus on high legibility of small-to-medium sized text on computer screens.”
Dana Library Hand, font created by Margo Burns inspired by a writing style recommended for card catalogues by 1899 “A library primer”.
Leon Sans, javascript based font allowing interesting transformations and effects (awesome leonsans demos).
Fontello, icon font generator.
Press Start 2P or Kongtext (retro game fonts)
Greta Grotesk (based on Greta’s hand lettered signs)
Hell vetica (helvetica twisted by messed up kernings)
Vollkorn (includes ligatures that can make your text as unreadable as early typeface…)
TT2020 (typewriter font with realistic irregularities built in)
Raybees (cutout-style font, with gyphs like a skull in place of brackets)
ztext.js (3D typography that works with any font)
Handwritten.JS (converts text into realistic handwritten image)

Random Interesting CSS

Wed, 29 May 2019 00:00:00 +0000

NES.css (retro game styling)
Bootstrap/386 (retro computer style)
Tufte CSS (Edward Tufte inspired layout with lots of asides)
1linelayouts

Machine Learning

Wed, 15 May 2019 00:00:00 +0000

Intro workshop for digital humanities group.

Machine Learning (ML) is the application of algorithms and statistical modeling to allow computers to “learn” from data to do a task (often overlapping or used interchangeably with Artificial Intelligence / AI).

ML tasks are broadly separated into supervised or unsupervised learning. Supervised learning tasks typically involve feeding the algorithm a labeled training data set which is used to build a model that can then classify unknown items, making inferences based on what it knows. Unsupervised learning tasks involve feeding unlabeled data to an algorithm that can identify patterns and clustering in the grouping.

The ability to learn from data is changing the approach to many computational tasks, such as OCR or NLP, putting the focus on curating training data sets rather than developing new software (see Andrej Karpathy, Software 2.0, 2017). However, computational techniques can also challenge our expertise in DH, stretching even stats experts ability to evaluate the validity of complex models (for fun, spurious correlations).

Natural Language Processing (NLP)

NLP is a family of techniques to analyze unstructured language data found in everyday speech and writing.

Historically, it wasn’t based in ML, but relied on manually identifying rules and patterns in human speech that could be parsed by code. For example, take a minute to play with ELIZA (1966), an electronic psychologist based in early NLP pattern matching.

The web has provided an explosion of unstructured text, making NLP a huge business as enterprise seeks to extract information from social media or create chat-bots to minimize labor costs. Typical tasks involve chunking/stemming, part-of-speech tagging, named entity recognition (NER), classification, and sentiment analysis. Speech recognition, OCR , and text-to-speech are also considered NLP tasks.

Demos:

IBM Watson Natural Language Understanding (API trained on web content)
- VPOD sentiment analysis (poetry sent to IBM NLU API for sentiment analysis)
Text-processing NLTK demos (free API, based on Python NLTK)
Book Visualizations Sandbox (Text and sentiment analysis with Hathi books. Shared on the Observable platform, a web-based code notebook for javascript)

Fun with Text Generators

Unsupervised deep learning neural network models? Can you collaborate with a machine algorithm?

Sunspring (Oscar Sharp, Ross Goodwin, Thomas Middleditch, 2016)
Janelle Shane, Darth Vader’s recipe (aiweirdness blog)
Train a GPT-2 Text-Generating Model w/ GPU For Free, google colab
GPT-2 code, recent press 1 2, 3, etc…
Talk to Transformer (ask it a question, like Q: What is Digital Humanities?)

YouTube API tips

Mon, 06 May 2019 00:00:00 +0000

YouTube is pretty handy for hosting video and audio to embed in your sites. Here are some tips to use the embeds and image API effectively.

Video Embed Options

When grabbing the embed iframe code from YouTube share button, I usually modify a few options. Some of these are documented in YouTube Player iframe reference, but not all or very well…

Privacy-enhanced mode: does not send data to YouTube unless users click play, is enabled by using domain www.youtube-nocookie.com rather than www.youtube.com in the iframe src.
Related videos: the option rel=0 asks YouTube to not show related videos. However, YouTube changed this option in 2018 and may still show related videos from the same channel (which is better than totally random stuff at least).
Modest branding: the option modestbranding=1 removes some YouTube branding from the iframe player.

This will look like:

<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/oacUgWXrqwc?rel=0&modestbranding=1" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

Image API

YouTube image API is used by a ton of sites, yet doesn’t seem to be documented anywhere by Google. Basically, you can get four sizes of the default thumbnail, or four smaller thumbnails from different points in the video using a YouTube ID. Find the YouTube ID by look at the v key parameter at the end of the video’s url or the last part of the “Share” url. For example, in

https://www.youtube.com/watch?v=oacUgWXrqwc

or

https://youtu.be/oacUgWXrqwc

the youtube id is oacUgWXrqwc. Use that string to replace < youtube id> in the api recipes below.

Default images:

thumb 120x90, https://img.youtube.com/vi/< youtube id >/default.jpg
medium quality 320x180, https://img.youtube.com/vi/< youtube id >/mqdefault.jpg
high quality 480x360, https://img.youtube.com/vi/< youtube id >/hqdefault.jpg
SD 640x480 (not available for all videos), https://img.youtube.com/vi/< youtube id >/sddefault.jpg
max quality 1280×720 (or 1920x1080?, not available for all videos), https://img.youtube.com/vi/< youtube id >/maxresdefault.jpg

Auto thumbs:

default image 480x360, https://img.youtube.com/vi/< youtube id >/0.jpg
alternate thumb 120x90, https://img.youtube.com/vi/< youtube id >/1.jpg
alternate thumb 120x90, https://img.youtube.com/vi/< youtube id >/2.jpg
alternate thumb 120x90, https://img.youtube.com/vi/< youtube id >/3.jpg

Note: you can use the domain img.youtube.com or i3.ytimg.com.

For more control, you can use YouTube Data API, but it requires a key to access.

Handy Jekyll Tips

Fri, 19 Apr 2019 00:00:00 +0000

random notes about Jekyll quirks

Add attributes to elements

Kramdown allows you to add additional attributes to elements with a basic syntax called Attribute Lists. Directly proceeding or following the block or inline element (as written in Markdown), add a {: the attributes, and close }. Multiple attributes can be added inside the brackets.

For example, to create <p id="example" class="red">Stuff</p>, use:

Stuff
{: #example .red }

To create <a href="https://example.com" target="_blank" rel="noopener">example</a>, use:

[example](https://example.com){:target="_blank" rel="noopener"}

Mix HTML with Markdown

In a Markdown file, all HTML is valid Markdown. However, Markdown inside HTML tags will be ignored.

For example, <div> # Header One</div> won’t render as an <h1>. However, with Jekyll’s kramdown, you can add the attribute markdown="1" to an HTML element to tell the converter to render the Markdown inside. Thus, <div markdown="1"> # Header One</div> will render as <div><h1>Header One</h1></div>.

Use variables to write Liquid expressions

Jekyll’s version of Liquid tweaks some of the functions to provide additional features. Sometimes it is handy to use a variable to actually write/modify a Liquid expression. For example, a page or site variable can be used to set up an include, basically Liquid inside a Liquid expression:

{% include {{ page.variable }} %}

Add YML front matter

Jekyll processes any file that has YML front matter on top. Any file that does not have front matter is just copied directly to _site as a static asset, thus you can not use any Liquid, etc. Sometimes you want to copy over HTML or assets from an old project and need to add YML header to each file. Use sed, like sed -i '1i---\n# \n---' file.html. For a batch:

for f in *.html; do sed -i '1i---\n# \n---' "$f"; done

Grab key / value when iterating over hash

If you have a YML hash like:

example: 
    key1: value1
    key2: value2
    key3: value3

You can iterate over the hash, in the loop access the key with [0] and value at [1], like:

{% for e in site.data.file.example %}
key: {{ e[0] }}
value: {{ e[1] }}
{% endfor %}

(see old Liquid wiki docs)

Python JupyterLab and Kernels

Thu, 18 Apr 2019 00:00:00 +0000

Mini workshop.

Intro
Survey, click here!
Discuss
Environments and Kernels
Notebooks

Conda

Package + environment management for Python (but can be used for other languages), conda docs.

Version and updates:

conda --version / conda info
conda update conda
conda update --all / conda update anaconda

Packages:

conda install [package]
conda remove [package]
conda list (all current packages + versions)

Keep in mind if you install something without using conda (i.e. pip), conda can’t update it.

Environments

conda create -n testenv
conda env list
conda activate testenv
conda deactivate
conda env remove -n testenv

conda env export --name myenv > myenv.yml
conda env create --file myenv.yml

Installing kernels

Python 2:

conda create -n py27 python=2.7
conda activate py27
conda install ipykernel
source deactivate

conda create -n mro_env r-essentials mro-base

Julia:

Download Julia and install
which jupyter
JUPYTER=$(which jupyter) julia
using Pkg
Pkg.add("IJulia")

Kernels list

Navigator

Anaconda Navigator is a GUI app to manage conda and install stuff…

Navigator docs
start on Linux: anaconda-navigator
create / manage / switch environments
manage packages, check versions
links to learning resources
launch apps in the correct environment

JupyterLab

It’s the future and it’s ready (news, blog). Notebook was designed around 2011–since then Python has been relatively stable, but web technology has changed rapidly… Jupyter Lab is an update using modern web technology,
If you have an up-to-date Anaconda install, it’s already installed
JupyterLab docs
fully backwards compatible (“Classic” Notebooks runs from the same server)
updated framework (built on JS)
more extensible
more IDE like (tabs and panes)
updated notebook features (drag & drop cells, collapse, context menu, more live preview formats and syntax highlighting)
code console (with notebook, text file, or codeblock in markdown > right-click to start console, shift+enter to run line or block)
Output view (right-click > “Create new view for output”)
conda install nb_conda
jupyter lab

nbviewer (easy way to share a static rendered version of a notebook from GitHub)
binder (open a GitHub repo of notebooks in a temporary executable environment, docs)

Library resources

R Resources

Sun, 03 Mar 2019 00:00:00 +0000

The R community seems like it is really taking off as a supportive, inclusive coding group in academics with less of the exclusion and barriers associated with traditional CS. I rarely use R, but want to expand my knowledge to provide better support on campus.

So here is random R links that I need somewhere to put before I forget…

Tutorials / learning:

R4DS Online Learning Community
Getting Started with R, R for the rest of us
RStudio Primers
Prelude in R
Software Carpentry Programming with R or R for Reproducible Scientific Analysis.

Reference:

R Graphics Cookbook, 2nd ed., Winston Chang.
R for Data Science, Grolemund and Wickham, 2017.
Text Mining with R: A Tidy Approach, Silge and Robinson. 2019.
Advanced R, Wickham, 2014.
Computational Historical Thinking: With Applications in R, Lincoln A. Mullen (work in progress).
Learning Statistics with R, Danielle Navarro (open course text/outline).

Articles:

Setup XPS 15 with Pop_OS

Fri, 25 Jan 2019 00:00:00 +0000

I recently got a Dell XPS 15 9570 (16gb ram, i7-8750H CPU) for work expecting it to have good support in Linux, considering other XPS laptops ship with Ubuntu. However, the massive (for a laptop) NVIDIA GeForce GTX 1050Ti with 4GB GDDR5 ensures installing Linux is not so smooth…

Update: Ubuntu 20.04 installs/works great on XPS 15. I have moved to Ubuntu 20.04, but most of this post is still relevant. The most recent versions of Pop!_OS have some unique gnome extensions, including an auto-tiling desktop option.

Any distro will boot into the live usb pretty well with the basic Intel graphics–however, you will eventually end up with a few issues trying to get NVIDIA to work. To simplify, I decided to use Pop!_OS, an Ubuntu-based distro from computer manufacturer System76. There is a few differences from Ubuntu, but most importantly Pop!_OS provides a special image specifically for NVIDIA machines so it is easy to get things working.

Note: I did not install as a dual boot, so I am not sure how smoothly that would go, but check the official docs.

Set up and install Pop!_OS

download Pop!_OS choosing the NVIDIA option.
Burn to a USB stick
Plug in your Pop!_OS USB
Boot XPS 15 and hit F2 as it boots to enter the BIOS menu
Enter BIOS:
- turn off Secure Boot (this is necessary for the NVIDIA drivers, and eventually if you want to use virtual machines)
- Settings > General > Advanced Boot Option, select “enable legacy option ROMS”
- SATA Operation, select “AHCI” (you need to turn “RAID” mode off because it is not supported by Linux, because its not worth while… If you are dual booting, check the steps on the windows side before doing this)
Save and exit BIOS
Reboot and hit F12 as it boots to enter boot option menu
Choose the first USB boot option
After a minute, it should boot into a live desktop with the Pop installer running. You can test out the live system if desired.
Follow the installer steps. It is a bit different than other distros because setting up a user takes place after install, during your first boot. Check the official docs. I chose to do a full disk encrypt (work computer), but keep in mind it will ask you for the encryption password (different than your user password) at every restart.
After installation is complete, reboot, and type in your encrypt password a few times
Follow the steps to set up a user account
Checkout your new system! See the intro docs and keyboard shortcuts for more info.

Updates

If you have used GNOME before, everything should be pretty familiar although it starts out more vanilla than an Ubuntu install. Click on Activities or hit Super (windows key) to bring up your menu.

You should run updates, which brings up the first noticeable difference from Ubuntu. The “update manager” is gone, instead updates are applied from the “Pop!_Shop” software center. Open Pop!_Shop, click on Installed, and if updates are available a button will appear to install them.

Graphics switch

Another unique Pop! feature: click the upper right for the power menu, then click the battery icon / status meter. This opens options to set power profile (performance / balanced / long life), or to switch between graphics cards (NVIDIA / Intel). Switching between graphics cards takes a minute and a restart–its not instantaneous…

Set up and customize desktop

Open Settings:

Settings > Background to get your wallpaper set.
Settings > Devices > Display to set up your monitor and turn on Night Light if you want a red shift in the evening.
Settings > Bluetooth, turn bluetooth off if you don’t use it, since it is a security risk.

Pop uses Ubuntu packages. Open a terminal and install these right away, sudo apt install:

gnome-tweak-tool (lets you customize the desktop)
chrome-gnome-shell (might be installed already, but enables gnome extensions, connecting the manager with Firefox or Chromium)
tlp tlp-rdw (advanced laptop power management, also see system76 battery)

Pop! comes with a lot of development essentials already installed (build-essentials, git, curl, gpg2, ufw). But you do have to enable firewall: sudo ufw enable.

Next, set up Firefox browser:

Preferences > Privacy & Security > check “Use a master password” and set.
Add GNOME extensions plugin, gnome-shell-integration
Add Firefox Multi-Account Containers, Facebook Container
Add adblocker such as uBlock Origin

Now you can add GNOME extensions, click the GNOME icon in the browser’s upper right side. Pop! comes with a few pre-installed, but here are a few more to add:

Caffeine, disable screen lock to watch movies or just not have your laptop locking all the time.
Dash to Dock, adds a highly customizable dock to your desktop. For an Ubuntu-like look, go to installed extensions page: click settings, turn off intelligent autohide, select panel mode, and make icon size smaller.
More suggestions, see GNOME on Ubuntu notes

Simplify git credentials

You can permanently store your Git credentials with your user account. Configure Git as normal:

git config --global user.name "Evan Will"
git config --global user.email "[email protected]"
git config --global core.editor "nano -w"

Install dependencies:

sudo apt install libgnome-keyring-dev

Build the helper:

cd /usr/share/doc/git/contrib/credential/gnome-keyring
sudo make

Configure git helper:

git config --global credential.helper /usr/share/doc/git/contrib/credential

Then the next time you need a Git credential, it will prompt for username and password as normal, but it will be stored going forward.

Note: newer versions of Git require a different manager, if libgnome-keyring-dev gives a permission error, use libsecret instead. I found this is necessary on Ubuntu 19.04.

sudo apt-get install libsecret-1-0 libsecret-1-dev
cd /usr/share/doc/git/contrib/credential/libsecret
sudo make
git config --global credential.helper /usr/share/doc/git/contrib/credential/libsecret/git-credential-libsecret

Issue with sleep

With Intel graphics activated, closing the lid for sleep or using the sleep icon from the power menu works correctly. However, XPS 15 does not seem to sleep correctly with NVIDIA activated, either it won’t go to sleep or it won’t wake up the screen…

Supposedly there is a way to fix this with boot options passed to the Linux kernel. If you were on Ubuntu, editing the GRUB options in theory should work. However, Pop! no longer uses GRUB, instead uses systemd-boot and kernelstub.

Ubuntu would look something like:

Open GRUB options, sudo nano /etc/default/grub, and edit the value of the GRUB_CMDLINE_LINUX_DEFAULT line, so it looks like:

GRUB_CMDLINE_LINUX_DEFAULT="quiet acpi_rev_override=1 acpi_osi=Linux nouveau.modeset=0 pcie_aspm=force drm.vblankoffdelay=1 scsi_mod.use_blk_mq=1 nouveau.runpm=0 mem_sleep_default=deep"

Then update GRUB: sudo update-grub

See dell-xps-9570-ubuntu-respin “xps-tweaks.sh” for scripts which inspired these settings.

You can do this on Pop (since all the Ubuntu stuff is still there), but it won’t have any effect, since it isn’t actually booting via GRUB!

Check your current set up with sudo kernelstub -p (mine says the kernel options are “quiet loglevel=0 systemd.show_status=false splash”). You could then add on the same kernel options as the GRUB config above, like:

sudo kernelstub -a "acpi_rev_override=1 acpi_osi=Linux nouveau.modeset=0 pcie_aspm=force drm.vblankoffdelay=1 scsi_mod.use_blk_mq=1 nouveau.runpm=0 mem_sleep_default=deep"

Honestly, I haven’t tried it… Since sleep works fine on Intel graphics, which I generally use for better battery life anyway, I will wait for some other brave soul to figure out the best boot options. Let me know if you do!

Add some apps

Languages:

Ruby: RVM set up
Python: Anaconda set up
Java: sudo apt install default-jre or default-jdk (this will be OpenJDK 11+, fine for apps like OpenRefine, if you need more specific versions search packages)
Node: use nvm (this seems to be the best method; install nvm, then nvm install --lts), or official NodeSource ppa, or repository version nodejs (which is out of date)

Search Ubuntu Packages or Pop!_Shop for software. App essentials:

A few extra fonts: fonts-firacode, fonts-noto
Chromium Browser, chromium-browser
Tilix terminal, tilix
KeePassXC
GIMP image editing, gimp

File types notes

Sun, 06 Jan 2019 00:00:00 +0000

notes clarifying plain text which is often mentioned in other notes…

Bits and Bytes

Bit = binary digit, i.e. 0 or 1. Byte = a grouping of 8 bits (normally) that are used together, which was historically just the right amount of data to encode text characters.

Hex = a convenient way to represent bytes as two hexadecimal digits, i.e. if you have groups of 4 bits, there is 16 possible combos from 0000 to 1111, represented in hexadecimal from 0 to F. Files can be opened in a “hex editor” to view the raw bytes represented as pairs of hexadecimal digits. For example, in UTF-8 A is 41 in hex, B is 42, Z is 5A, etc.

Text vs. Binary

Generally, there is a distinction between two broad types of computer files: text or binary.

Text files contain a series of bytes intended to represent text characters organized in lines (e.g. a, B, space, tab, line breaks, etc.). It can be opened with a text editor to see the characters.

Binary files contain bytes that are NOT character codes, data that is meant to be interpreted as something other than characters. It will require software (other than a text editor) that can correctly interpret the bytes. For example, a JPG image, MP3 sound file, or a ZIP compressed folder. A binary file can be represented in a text file by binary-to-text encoding standards, such as Base64, where the bytes are represented by text characters. The text will be gibberish of course, but can be converted back into the binary format to retrieve the original data.

Usually, when talking about text files, we mean plain text. It is defined by Unicode standard as:

“Plain text is a pure sequence of character codes; plain Unicode-encoded text is therefore a sequence of Unicode character codes.”

Plain text files are text files, i.e. contain only characters like a, 1, <, !, etc. Some characters might be “hidden” control characters, such as tabs and line breaks.

In some contexts you will hear a distinction between plain text and rich text, i.e. just the characters/words vs. text with information about the style/presentation (fonts, colors, sizing, etc). However, “rich text” is often written in plain text, e.g. HTML, LaTeX, Markdown, or XML. So in this distinction, hello world is plain text, but <h1>hello world</h1> is rich text written in plain text. On this blog I call text files “plain text”, and don’t use the distinction with “rich text” in this way.

Raster vs. Vector

There is a broad distinction between two types of image file types:

Raster image files contain data that describes the individual pixels that make up the image. This are often captured by a digital camera or scanner. Common formats include JPEG, PNG, TIFF, etc.

Vector image files contain data that describes the geometry/geometric features that makes up the image. Since digital screens are made up of pixels, vector images are rendered into a raster image when viewing–however, because the vector image is made of points, lines, curves, etc. it can be scaled to any size without loss of quality. Common formats include SVG, which is written in XML.

Demos

Open these all with a text editor:

text file (text)
html file (text. with markup)
docx file (binary. A zipped folder of XML components)
unzipped docx (use unzip or right-click to extract contents which are plain text files)

Open this “dogtest” image in various formats with a viewer and text editor:

png (raster, binary, lossless compression)
jpg (raster, binary, lossy compression)
svg (vector, text. Open with a text editor to view the XML, edit the numbers to change the image)
pbm (raster, text. Open with a text editor to see black and white pixels represented as 1 and 0, edit to change the image)

Setup Jupyter with Py 2, 3, and R

Sun, 06 Jan 2019 00:00:00 +0000

Note: this is out of date and due to new licensing terms, I no longer suggest using Anaconda.

Jupyter / Jupyter Notebook is a great tool for teaching code and exploratory, iterative coding.

You can quickly demo it online at Try Jupyter!

Jupyter was originally developed for Python, but it now supports kernels for many languages. With multiple kernels installed, you will be given a choice to launch notebooks using any of those languages. Thus, Notebook becomes even more useful with both Python 2 and 3 installed (keep in mind Python 3 is current, 2 is legacy). And you may as well add R while you’re at it!

1. Install Jupyter via Anaconda

First, Jupyter project and I suggest you install Python 3 via the Anaconda distribution. Anaconda gives you a huge package of scientific Python libraries pre-installed, plus conda, a very handy management and virtual environment tool (also comes with Anaconda Navigator GUI, which is kind of annoying, and Spyder IDE). Once you install Anaconda Python 3, you automagically have Jupyter with the IPython3 kernel ready to go.

Download latest version of Anaconda Python 3
Run installer, but be sure to say yes to “add Anaconda to my PATH”.

If you already have Anaconda installed, be sure to update before adding the additional kernels. Anaconda can be completely upgraded using conda update conda, then conda update anaconda.

2. Add Python 2 kernel to Jupyter

Second, add the Python 2 kernel using a virtual environment:

open a terminal and create a new Python 2 environment: conda create -n py27 python=2.7. This will take a minute to install the Python 2 dependency.
activate the environment: linux source activate py27 or windows activate py27
install the kernel in the env: conda install ipykernel
close the env: source deactivate

3. Add R kernel to Jupyter

Third, add R via Conda. Conda R-essentials package includes a bunch of popular R packages, including the Notebook IRKernel.

Note: Anaconda gives you the choice of using MRO or normal R. Use conda packages mro-base or r respectively.

If you want R to be generally available, simply open a terminal and install R, R-essentials, and RStudio (optional, more notes): conda install -c r r-essentials rstudio

If you would rather keep R in a environment, open a terminal and create a new R env: conda create -n mro_env r-essentials mro-base. The R installed in this env will be available to Jupyter Notebook.

Start Notebook

Open a terminal and type: jupyter notebook

The Notebook interface will open in your browser (ignore the server back end running in the terminal). On the right side of the Notebook, click the “New” button. You should have the options for Python [default] (i.e. conda py 3), Python 2, and R. Pretty handy for teaching and learning!

To shut down Jupyter, close the browser window, then Ctrl + C in the terminal host.

Extensions

Conda has a few other addons available for Jupyter. Check the extension documentation to learn more.

BeakerX (newish project recently converted into an extension of Jupyter allowing multiple languages in a single notebook and other enhancements. Can be installed via conda)

nbviewer (easy way to share a static rendered version of a notebook from GitHub)
binder (open a GitHub repo of notebooks in a temporary executable environment, docs)

JupyterLab

JupyterLab is the next generation of Notebook that has been rolling out since about Feb 2018 (news). Designed as a more complete IDE-like environment, Lab still fully supports notebooks, but provides more advanced features. If you installed Anaconda, you already have Lab ready to go, and all your environments will be available.

Open a terminal and type jupyter lab to start it up.

JupyterLab is actually available when you run jupyter notebook too, from the base url, something like http://localhost:8888/tree, replace the directory with /lab.

Other Notebooks

Notebooks are gaining attention for integrating code and publication, opening possibilities for interactive sharing and visualization.

SageMath (math focused notebook platform, an “open source alternative to Magma, Maple, Mathematica, and MATLAB”. A solid project that has been around fsince 2005, but hasn’t gained much popularity.)
Apache Zepplin (Java based notebook particularly useful for dashboards and Spark cluster integration, i.e. big data)
nteract (desktop app or Atom package built on top of Jupyter kernels to improve/simplify notebook experience)

Others are running in the cloud to lower barriers to getting started and promote reproducibility:

Colaboratory (Jupyter environment hosted by Google designed to run TensorFlow without installing anything, with your notebooks stored in Google Drive)
Observable (JavaScript based data visualization for the web)
Code Ocean (a variety of notebooks and IDE available)
COCALC (was SageMath Cloud, now has Jupyter Notebook with lots of kernels, a LaTeX editor, and more)
Iodide and Pyodide (web focused notebooks implemented in browser, in alpha development)
Stencila (word processor + spreadsheet + code)
Azure Notebooks

Issues on Linux

When installing on Fedora I have run into issues where some applications no longer work because they are trying to use Anaconda rather than the default system Python (I haven’t had this happen on Ubuntu). The installer adds a line to your .bashrc in order to make Anaconda available on your path. Open sudo nano ~/.bashrc, and look for a line like:

# added by Anaconda3 installer
export PATH="/home/username/anaconda3/bin:$PATH"

One quick fix is to flip PATH and Anaconda, so your system Python will be found first by default. Change the .bashrc line above to export PATH="$PATH:/home/edog/anaconda3/bin", then reboot. Your applications will correctly find the system Python, but you can still easily access Anaconda’s conda, ipython, and jupyter. Keep in mind, if you type python in the terminal, it will be the system Python, not Anaconda.

Update GitHub Fork

Tue, 04 Dec 2018 00:00:00 +0000

Forking a repository is a feature unique to GitHub (i.e. it is not a Git function). A fork is a complete copy of a repository that retains a link to the original. They are used in a similar way to Git Branches, allowing you to experiment and make changes independent of the main/master version which can then be merged back into the source. This supports a workflow where anyone can offer contributions to open projects via pull requests.

However, it is often a bit mystifying how to keep your fork up-to-date with the original. A simple solution is to delete your stale fork and then fork a new one–but there is (of course) a way to update safely.

Instructions below use main as default branch, if you have an older repository the default may be master instead

Sync fork on commandline

To sync your fork from the original repository is two steps:

Setup the upstream remote (only have to do this one time)
- check current config: git remote -v
- add upstream remote (use the clone link from the original repo): git remote add upstream https://github.com/ORIGINAL_OWNER/ORIGINAL_REPOSITORY.git
Sync fork
- git fetch upstream
- git checkout main
- git merge upstream/main
- You’re up-to-date locally, then update your fork on github, git push

Sync fork on GitHub Desktop

Click current branch button in top menu (this should bring up a branch panel)
At bottom of the branch panel click “choose branch to merge with main” (this should open a dialog box)
Click “upstream/main” (this should be the original repo that you forked)
Click “Merge upstream/main” (this will fetch the updates in the original repo and attempt to merge them into your current branch)

Sync fork on GitHub web interface

There is also a weird work around to do it via the GitHub web interface:

Go to the original repository (not your fork).
Click on “New pull request”
On “compare changes” page, click the little text that says “compare across forks”.
Select your fork from the “base fork” drop down (this means you are pulling code from the original and sending it to your fork), if all is good it will say “Able to merge”
Add a PR message and click green “Create pull request” button.
Your fork now has a PR, click “Merge pull request” to complete.

ExifTool Notes

Tue, 27 Nov 2018 00:00:00 +0000

To read, write, and manipulate metadata embedded in image files you need ExifTool (p.s. can also read/write metadata on many other file formats such as PDF, videos, music, and documents). Truthfully, nothing else works reliably.

Install

ExifTool is a Perl library and commandline tool. Check the install docs for full information, but basically on Windows use the “stand-alone executable”, Mac use the “MacOS package”, and on Linux use your distro repository.

Windows:

Download the “stand-alone Windows Executable” from the ExifTool page (this will be a .zip file e.g. “exiftool-11.99.zip”).
Unzip the downloaded file.
Inside you will find a file named exiftool(-k).exe. Rename it to exiftool.exe.
Add it to your PATH:
- If you use Git Bash, find your Git Bash root directory by typing cd / then pwd -W. In your Git Bash root, copy exiftool.exe into the bin folder (see my note on adding stuff to Git Bash for more detail). ExifTool will only be available on Git Bash, but this install method does not require admin access and I would suggest only using Git Bash with it anyway.
- Alternatively, you can copy exiftool.exe into a directory on the Window’s PATH, i.e. C:\Windows. ExifTool will be available on CMD and GitBash.

Linux:

install ExifTool as the Perl library, e.g. sudo apt install libimage-exiftool-perl.

Use

To use ExifTool, start with the command exiftool, add some options, and the file name of an image. Type exiftool to printout the full manual page.

Read Metadata

The most basic command is to read all metadata for an image:

exiftool test.jpg

However, Exiftool has batch options built in, so you could replace the filename with a selector such as *.jpg to read multiple images.

From there add Options and/or Tags to the command.

Options typically change the tool’s output. Option examples:

-v, the Verbose Option adds more technical detail to the output: exiftool -v test.jpg
-T outputs the data as a tab delimited table: exiftool -T test.jpg > table.txt
-csv outputs as csv and automatically adds a “SourceFile” column with the file name: exiftool -csv *.jpg > metadata.csv
-X outputs RDF/XML format: exiftool -X test.jpg > metadata.xml
-ee reads metadata for embedded files in addition to the main file, e.g. images combined into a PDF: exiftool -ee test.pdf
-a (allow duplicates) and -u (allow unknown tags) extract metadata that might otherwise be hidden: exiftool -a -u test.jpg

Tags correspond to embedded metadata elements and are used to read or write specific values. Metadata elements are given a “Tag Name” (and machine readable “Tag ID”) that ExifTool can take as a commandline argument by adding - in front. To find the notation explore the Tag Names index.

To read a specific tag, include it as an argument. For example, exiftool -Make test.jpg will output the EXIF metadata for camera/scanner make.

A batch of GPS tags can be extracted using:

exiftool -gpslatitude -gpslongitude -csv *.jpg > locations.csv

Note, some tags are part of a larger “group”, such as IPTC or XMP, and may have a “namespace”. The tag names are prefixed with the group name, for example, -XMP:Description or -IPTC:Source. Namespaces are added to the group name, for example, -XMP-dc:Source.

Write Metadata

Writing metadata follows the same pattern as reading, but adds a value to the Tag. To write a specific tag, include the tag argument followed by a = and value.

For example, to build up the command flag we want to add XMP metadata (-XMP), from Dublin Core namespace (-dc), in the tag Source (:ProjectionType), with the value of “Example source” (="Example source"). Thus to add/change that one metadata value, the full command would be: exiftool -XMP-dc:Source="Example source" test.jpg.

Writing examples:

exiftool -Title="New title value" test.jpg will delete the current EXIF title and replace it with “New title value”.
exiftool -Title= test.jpg will delete the current EXIF title, i.e. replacing the value with nothing.
exiftool -all= test.jpg will delete all current metadata except the most basic technical EXIF.
exiftool -XMP-dc:Source="Uni, Uni Archives, Wonderful Collection" test.jpg will add XMP Dublin Core metadata.
Multiple tags can be written simply by adding more flags, exiftool -Title="New title" -XMP-dc:Source="Uni, Uni Archives, Wonderful Collection" test.jpg

Note, although there are lots of possible ways to embed metadata, not many are actually readable to commonly used applications. If you want the metadata to be visible in Window’s file properties stick to common tags such as Title, Authors, and Copyright. Full XMP DC elements are not commonly read. IPTC metadata has a ~30 character limit.

Use for Batches

If you want to add the same metadata to a large batch of files, it is most efficient to use ExifTool’s built in batch processing. For example, adding a wildcard * and file extension .tif will add the metadata tags to all TIF files in a folder:

exiftool -Title="Example Collection" -Copyright="Photo courtesy of the Example Collection, University of X." -XMP-dc:Source="Example Collection, SPEC, University of X." *.tif

If each file requires individualized metadata, it is possible to use a Bash loop, although this is much slower. For example, this loop uses the file name (assumed to be based on an identifier) to add an identifier to the metadata:

for f in *.tif; do exiftool -Title="Example Collection, ${f%.tif}" -Copyright="Photo courtesy of the Example Collection, catalog nos. ${f%.tif}, SPEC, University of X" -XMP-dc:Source="Example Collection, catalog nos. ${f%.tif}, SPEC, University of X" "$f"; done

Guidelines for Digitized materials

FADGI, Minimal Descriptive Embedded Metadata in Digital Still Images (PDF)
FADGI, Embedded Metadata in TIFF Images

OpenRefine for Metadata

Mon, 26 Nov 2018 00:00:00 +0000

Outline:

Import metadata:
- character encoding (UTF-8!)
- import options (separator, batches)
- CONTENTdm TSV parsing issues: uncheck the option Use character " to enclose cells containing column separators
Explore metadata:
- facets (from column menu)
- text filter (from column menu)
Clean up metadata:
- editing facets / cells (hover on value)
- trim whitespace (from column menu)
- Star / Flag, and facet (all column menu)
- remove rows (all column menu)
- reorder columns (all column menu)
- split into multiple columns (from column menu)
Clean up multi-valued cells (e.g. Subjects):
- split multi-valued cells (from column menu)
- trim (from column menu)
- cluster (from column menu)
- join multi-valued cells (from column menu)
Transform using GREL:
- add text, value + " something new"
- find & replace, value.replace("old","new")
- get values from other cells, cells['column_name'].value
- dates: GREL dates, value.toDate().toString('yyyy-MM-dd')
Export metadata:
- CSV export
- templating

Reference:

GREL
Clean Your Data workshop
Fetching and Parsing Data from the Web with OpenRefine tutorial
Handy OpenRefine Functions to Remember

Get started with Linux!

Thu, 08 Nov 2018 00:00:00 +0000

Please check out Try Linux, an updated version of this workshop.

MILL workshop:

Want to get started with Raspberry Pi or Linux on your laptop? This workshop will introduce the most popular operating system in the world! Although most people are more familiar with Windows or Mac, Linux OS runs the vast majority of web servers, super computers, smartphones (Android), TVs, and other electronic devices. You can also use desktop versions for free on your laptop or home computer! Come get an introduction to free and open source operating systems, and see desktop versions of Linux in action on Raspberry Pi and laptops. We will get oriented to the basics of the desktop and learn how to install it on a computer.

The following notes provide a TON of information about getting started with Linux. But first, take a deep breath… don’t get overwhelmed by all this information!

The barrier to getting started with desktop Linux on your own computer is much lower than you expect. Choose a Linux distribution (I suggest Ubuntu to start), create a live USB, plug it in to your computer, and voilà–you can try out Linux without having to install anything!

Learn all the details from these notes:

Intro to Linux, learn about how to choose a Distro.
Burn USB, learn how to create a Live USB install media.
Boot USB, the tips and tricks to boot a Live USB.
Demo time (explore the “live session” from the USB. connect to the wifi, try the apps, just don’t click install!)
Command Line demo (optional)

Introduction to Linux!

Tue, 06 Nov 2018 00:00:00 +0000

Mini workshop:

Want to get started with Raspberry Pi or Linux on your laptop? This workshop will introduce the most popular operating system in the world! Although most people are more familiar with Windows or Mac, Linux OS runs the vast majority of web servers, super computers, smartphones, TVs, and other electronic devices. You can also use desktop versions for free on your laptop or home computer!

What is Linux?

Linux (pronunciation) is a family of open source operating systems (OS). You might hear “Unix-like” or “*nix”, since Linux is part of a group of OS descended from the early research computing system UNIX.

An OS is made up of a many components, basically:

Bootloader: starts the low level processes necessary to boot an OS, most linux distributions use GRUB.
Kernel: manages the hardware and low level software components, coordinating the complete OS. This is actually “Linux”.
Shell: text-based command line interface.
Desktop environment: graphical interface .
Applications: individual programs run from the desktop or shell.

The core of Linux, the kernel, is developed by the Linux Foundation and original creator Linus Torvalds, with contributions from thousands of people and corporations around the world. Linux operating systems are usually shipped as a complete package of components called a distribution, or “distro”. There are hundreds to choose from, each with different philosophies, strengths, and use cases.

Why Linux?

Linux core is Free Software, meaning cost-free and freely licensed, giving you freedom (as in freeeeeedoooom)! You can control, inspect, modify, copy, and share the OS. But for most users, that is not necessarily the most compelling reason to use Linux. Instead they just appreciate a solid, powerful, and user-friendly OS. Linux provides security, privacy, stability/reliability, customization, and cost benefits, in addition to access to a huge ecosystem of open-source applications and services.

Linux runs the vast majority of web servers (95%+), super computers (all of the top 500), smart phones (80%+), Chromebooks, IoT, and single board computers. So if you want to use a VM in the cloud, do high performance computing, or tinker with a Raspberry Pi at home, it’s helpful to know Linux. But it also makes a great desktop OS for your personal computer! Thus, there are many practical and ideological reasons to use Free and Open software.

For an in depth introduction, try the free edX course from the Linux Foundation, Introduction to Linux.

Bloatware, adware, viruses, oh my! Many people are sick of the shady commercial practices of Windows, Apple, and PC manufacturers that put user security and privacy at risk. Proprietary OS limit your rights and control as a user. Linux distros are viable alternatives!

Distributions

Surf Distrowatch to browse the latest distro news! Part of the fun of getting started with Linux is choosing a distro, but there are so many it can also be very overwhelming. Most distros are a package of technical low-level stuff, plus a desktop and selection of basic applications. Some are focused on running servers, others on desktop users.

Here are some things to consider:

Out-of-the-box experience. Do you want it to just work or are you willing to spend time customizing? Do you want all the applications pre-installed (openSUSE) or minimalist (Arch)?
Community. Do you want to search online and find hundreds of answers to your questions (Ubuntu), be part of a values driven community (Fedora), or get involved in a smaller forum of deeply invested enthusiasts (Arch)?
Repositories. Some distros have huge repos of applications (Debian), making installing new software secure and easy. Some are more up-to-date than others (Fedora). Others are more DIY (Arch). Some distros package only free and open software that meets their community guidelines (Fedora); others include commonly used, but non-free software, such as MP3 encodings (Ubuntu).
Ecosystem. Do you use a Linux server at work or some hobby project? Choose a distro from the same family to make life easier.
Hardware support. Unlike Windows, with Linux you don’t have to install a bunch of drivers to get everything to work. However, some distros support a huge range of legacy hardware, while others have better support for the newest hardware built in. It is often difficult to run high performance graphics cards with out specific distro support. Distros have different versions of the Linux Kernel–the newest have upgraded features and performance baked in, but have less of the kinks worked out.
Updates. Do you want to install an OS that will be supported for a LONG time so that you won’t have to upgrade or do you want the latest-and-greatest? Distros typically follow either a rolling or fixed release. Rolling release distros have a continuous stream of updates, you always have the latest software and never have to upgrade to a new version (but may encounter some instability). Fixed release are more traditional and stable with incremental versions that require a large upgrade every so often. Fixed releases are often broken into latest and LTS (long term support)–if you want to install once, and use the system for years without needed to upgrade, choose an LTS. For example, Ubuntu normal releases are officially supported for nine months, LTS for five years.
Installer! In the past, Linux distros were confusing to install, a major barrier to adoption–Ubuntu really changed that, and some installers are still friendlier than others.
- Secure Boot UEFI. If you have a newer computer, it will have Secure Boot UEFI which is only supported by the bigger Linux distros (Ubuntu, Fedora, SUSE)–otherwise you will have to tweak your UEFI / BIOS to get the installer to work.
- Dual booting. Do you want to keep windows? Some installers make it easy (Ubuntu).

My easy suggestion: use Ubuntu or an Ubuntu flavor. The installer is easy to use (supports easy dual booting with Windows) and hardware support is great. It also has the largest beginner and intermediate user community, so help is always easily at hand via blogs, wikis, and forums–if you search for general Linux help majority of answers are about Ubuntu. If a piece of software provides Linux instructions or packages, it will include Ubuntu.

But check out the Major Distro Families below for MORE suggestions!

Desktop Environment

Next, you will have to choose your desktop environment (DE)–the graphical interface you will be staring at all day! Unlike windows, mac, or chrome, with Linux distros you have a choice of desktops. It is separate from the core of the distro and can be installed independently.

Generally, you will choose a package pre-bundled with a DE, most distros will give you several options, with GNOME, KDE, and Xfce being most common. Distros often tweak their versions of the desktops, so the final look, default software, and features vary a bit. Just take a look at their screenshots and decide!

But here are a bunch more details…

Things to consider:

Weight. How resource intensive is it? Slicker desktops (GNOME, KDE) require modern hardware to run well. If you have an old computer or low spec hardware, go light (Xfce, LXqt)!
Configurability. How much do you want to tweak the look and feel? Do you want it to just look good out of the box?
Unique vs. traditional. Do you want something different (GNOME) or are you nostalgic for the old days (Cinnamon)?

See Major Desktop Environments below for lots more detail…

64 or 32

One last choice you might have to make is 64 or 32-bit? Many distros have started to phase out 32-bit support, but you may still encounter the choice to download “64-bit” / “x86_64” / “AMD64” versus “32-bit” / “i386” / “i586”. If your computer is still running, it is likely new enough to be 64-bit, so I would suggest ignoring 32-bit options unless you know your hardware requires it. Some distros now also support ARM processors, such as found in Raspberry Pi, so you may encounter an option like “ARMv7”.

Try and Install!

The best way to learn Linux is to try it.

When you buy a Windows, Chromebook, or Mac computer, the operating system (OS) comes pre-installed. However, if you are interested in running Linux or other open OS, you will usually have to install it yourself.

Most modern Linux distros are released as an ISO disk image, usually a “live cd” or “live usb”. This means you can download an ISO, burn it to a USB stick, and then boot up into the live desktop without actually installing or changing your hard drive!

Give it a try with these instructions to burn an ISO and boot a live USB.

Once you try it out, install it!

The live session will have an app to permanently install the OS on your machine. Most installers are graphical and user-friendly. Here is Ubuntu’s official install tutorial.

Alternatively, if you really don’t want to even download anything, you can visit Distrotest. Distrotest will boot up a virtual machine with your disto of choice and open a remote desktop window right in your browser. The performance is slow (depending on your connection and the VM) and not exactly like using real hardware, but can be a nice option to quickly check out a new OS.

As another option, you can install VirtualBox on a computer to try out any distro in a disposable virtual machine. The performance will be very limited, but it is an easy way to test something out or see what the install process looks like–and grab screenshots for tutorials or debugging.

Manufactures such as Dell and Lenovo are offering more computers with Linux pre-installed, and Linux specific companies such as System76 are booming. More manufactures offering devices means better hardware support is being built into the Linux kernel.

Basics

If you have ever used a computer, almost everything will be familiar! No big deal. Here are a few things slightly different from Windows:

User accounts and passwords. You are always logged in as a user and require a password to make any changes to the system (Windows tends to hide this).
File system, home directory. Linux uses a file system that Windows can’t read and is presented differently. On Windows the root is the drive letter C:\. Linux presents all drives as a unified file system, with the root at / and your user directory at /home/username (Linux doesn’t use Windows back slashes \).
Updates. Updates are pushed out regularly keeping you secure and fixing bugs–updates are much more common than on Windows, yet usually less of a hassle. The updates cover the OS and all applications installed from a repository. First, the catalog of all software in the distro’s repositories is updated, then updates that apply to your installed software (and dependencies) are downloaded. There is usually an updater application, but you may need to start it manually.
Software center. Applications can be easily and securely installed from your distro’s repository. There is usually an GUI app to find and manage software curated by the distro. Recently distro independent application packaging frameworks are supplementing or replacing traditional software repositories. The main frameworks include Snap (Ubuntu, Snapcraft), Flatpack (Flathub), and Appimage. These formats simplify software distribution, as each package has all dependencies fully self-contained (and often sandboxed).
Terminal. Command line is really handy! See this mini-workshop.

Major Desktop Environments

The major desktops can be divided by their “widget toolkit” used to create the GUI elements and default applications. This is probably the last time you will ever hear “widget toolkit”, but you might hear something like GTX theme or Qt application. The details aren’t important, but you will see some family resemblance. All desktops can support both GTX and Qt applications, but software in the native toolkit will have more closely integrated and themed.

Major desktops:

GTX based
- GNOME. modern, minimalist-style, unique “Activities Overview” (no start menu), extensions add functionality, and themes easily change the look.
- Xfce. lighter weight, simple, stable, traditional linux desktop. Offered as the default of many distros, often highly customized.
- Less common:
  - Cinnamon. more traditional desktop with start menu (get newest version with Mint).
  - Budgie. slick, GNOME-based (get newest version with Solus).
  - Pantheon. polished, GNOME-based (developed by ElementaryOS)
  - MATE. traditional, lighter weight, based on an old version of GNOME.
  - LXDE. very light weight, basic features (development has shifted to LXQt instead, so check out LXQt instead).
  - Historic note: ~~Unity~~. Until version 17.10, Ubuntu shipped with a unique desktop. However, Ubuntu’s GNOME theme emulates the Unity look.
Qt based
- KDE. very configurable, complex, windows-like (get newest version with Neon).
- LXQt. very light weight, basic features (sibling of LXDE).
- DDE. Polished desktop of Deepin linux (Chinese focused), recently becoming available in other distros.
Other
- Tiling window managers (the traditional desktop metaphor is replaced by automatically dividing the screen with your open application windows, controlled by key commands, for a simple and efficient desktop. There is a bit of a learning curve!) Checkout xmonad or i3. For any easy intro, checkout Regolith Linux which combines i3-wm with Ubuntu.

Most desktops support themes which make it easy to change the look and feel. For example check out the themes page at OMG! Ubuntu!.

Major Distro Families

Below is a basic introduction to the major linux families, with a focus on distros with desktop friendly versions.

Debian

One of the oldest active linux distros, Debian has extensive stable repositories that are the base of MANY other distros. This includes Ubuntu, developed by enterprise software company Canonical based in the UK. Uses apt configuration and .deb packages.

Debian, entirely free software released by the Debian Project. Debian packages.
Ubuntu, the most popular desktop linux, beginner friendly. Ubuntu Packages and Snapcraft.
- Ubuntu flavors, Ubuntu spins featuring alternative desktops.
- Linux Mint, one of the most popular and beginner friendly, featuring the Cinnamon desktop.
- Elementary OS, stylish mac replacement, unique Pantheon desktop.
- Pop!_os, customized Gnome desktop with some unique addons, built by computer company System76 focused on developers.
- KDE Neon, cutting edge KDE on Ubuntu base.
- Zorin OS, polished windows replacement.
- Linux Lite, focused on easy to use, windows replacement, with Xfce.
- Bodhi Linux, minimalist, lightweight distro with unique Moksha desktop.
- Peppermint, ChromeOS-like cloud focused minimal desktop.
- GalliumOS, Xubuntu based replacement for Chromebook hardware.
Raspberry Pi OS, official distro of Raspberry Pi, unique Pixel desktop (some other distros offer specialized spins for Pi as well)
Endless OS, simplified, education focused system designed for offline use, by computer company Endless.
deepin, Chinese based distro with unique desktop and apps.

Fedora / RedHat

Based in the USA, Red Hat is one of the largest open-source enterprise software companies, providing support, development, and Linux OS to major corporations. It is a good choice to learn if you are interested in working with enterprise environments. Uses yum or dnf configuration and .rpm packages.

Fedora, open project sponsored by Red Hat with a focus on free-software. Considered fairly cutting edge with two new versions each year, a testing ground for RHEL, yet stable enough for most users. Fedora Packages.
- Fedora Spins, alternative desktops.
- Fedora Labs, specialized distros from science to music.
- Sugar on a Stick, very unique kid focused learning platform, packaged to run on a live USB. Learn more at Sugar Labs.
Red Hat Enterprise Linux (RHEL), an industry standard, non-free.
~~CentOS~~, free community supported RHEL. Once very popular, was unexpectedly discontinued in 2020 during Red Hat corporate changes.
~~Scientific Linux~~, free RHEL-based OS developed by Fermilab for high performance scientific computing environments (is now being phased out in favor of CentOS).

SUSE

Germany based enterprise FOSS company. Uses ZYpp or YaST for configuration and .rpm packages.

openSUSE, developer focused with a rolling release option, Tumbleweed. Note: openSUSE default install ISO are not live images and can not be used to try without installing. They provide a separate live image, but it is not complete and cannot be used to install. openSUSE Packages.
- GeckoLinux, openSUSE spin focused on being user friendly.
SUSE Linux Enterprise, commercial distro with focus on mature and stable server environments.

Arch

An independent, community built family with rolling release following a K.I.S.S. and DIY philosophy. Not traditionally user friendly. Uses pacman for configuration. Archlinux packages.

Arch
Manjaro, user friendly Arch for beginners.
EndeavourOS, intended to build user-friendly installer and community for Arch.

Others

Independent. Some distros go it alone. For example:

Solus, up-and-coming independent distro developing Budgie desktop.
Qubes OS, personal security focused distro with unique architecture that isolates all software in separate VM-like containers. Major distros have also started offering fully containerized OS versions for max security and stability, such as Ubuntu Core and Fedora Silverblue (see explainer article).
Alpine Linux, minimalistic, security focused distro from Norway, commonly used as basic server image or as base for containers.
Linux From Scratch, build everything yourself, step-by-step DIY!

Portable. Some linux distros are specifically designed to be portable, i.e. you always use them from a USB stick and do not install to a hard drive. This makes them simple, fast, and secure. For example:

Android. Based on Linux, Android is the most popular smart phone OS in the world.

Android-x86, port of Android that can be run on a laptop.

Entertainment. There are many specialized minimalist distros that act as entertainment centers, usually run on a Raspberry Pi or similar device.

music player: Volumio.
media center: OSMC or OpenELEC.
gaming: RetroPie or SteamOS.
NAS: FreeNAS or EasyNAS.

Utilities. Some specialized distros are basic utilities used to work on a computer where the main drive or OS maybe corrupted.

GParted Live, partition and disk utility.
Clonezilla, disk imaging/cloning deployment or recovery.
DBAN, disk data destroyer.

Other open OS

There are other open-source OS out there, Linux is just the most popular.

BSD: a family of OS descended from Berkeley Software Distribution Unix. Considered rock-solid stable and secure for servers, with desktop options FreeBSD and TrueOS.
OpenSolaris / illumos / SunOS: a family of Unix-like OS descended from proprietary enterprise systems, currently openly developed by the openindiana community.
Haiku, based on BeOS, an early competitor to Mac.
KolibriOS, minimalist, tiny OS.
Redox OS, newly developed OS written in Rust language.

Resources

Julia Evans Drawings and Zines are an approachable and amusing way to learn about advanced linux topics.

Commandline notes

Tue, 30 Oct 2018 00:00:00 +0000

Just some general notes to remember helpful things on the command line…

Variable remove pattern

In a bash loop you often want to replace a part of a filename with something else. There is two easy functions to remove filename/variable parts:

Remove from front: ${var#pattern}
Remove from back: ${var%pattern}

For example, lets say you have a bunch of files with following the pattern front-[id number]-back.txt (front-10-back.txt, front-192-back.txt, etc.).

To rename while removing the front: for f in *.txt; do mv "$f" "${f#front-}"; done

To rename while removing the back and changing the file extension: for f in *.txt; do mv "$f" "${f%-back.txt}.md"; done

However, you can’t use both a front and back remove at the same time. To get around that, use an extra variable. To combine the other two commands:

for f in *.txt; do NAME="${f#front-}"; mv "$f" "${NAME%-back.txt}.md"; done

If you have spaces in the pattern, escape them. E.g. name with space use something like ${f#name\ with\ space}.

You can also use the syntax ${var/pattern/replacementstring} to replace things in the filename. If the “pattern” chunk starts with an extra /, then it replaces all matches (not just the first). This way you could remove all spaces from filenames in a directory and replace them with underscore like: for f in *\ *; do mv "$f" "${f// /_}"; done

Change String Case

Bash has a built in feature to convert the case of strings:

^ convert 1st character to uppercase
^^ convert whole string to uppercase
, convert 1st character to lowercase
,, convert whole string to lowercase

This can be helpful for renaming all files with lowercase filenames and extensions. To do the renaming by copying to a new folder “renamed”:

for f in *.*; do cp "$f" "renamed/${f,,}"; done

This could also be done using the older tr utility using a much more complicated command:

for f in *.*; do cp "$f" "renamed/$( tr '[:upper:]' '[:lower:]' <<<"$f" )"; done

Combine CSVs

Remove headers and combine all files into one:

tail -n +1 *.csv > all.csv

Then, each file will be separated by ==> filename <==. Open in OpenRefine, use separator ==>, then use <== to create filename column.

Or use a loop:

for f in *.txt; do cat $f; echo "separator"; done > combo.txt

Count files of a specific type

Sometimes you need a count of how many files of a specific type are in a folder including sub-folders. A quick method is to use find and wc to get a count. In the directory you want to count:

find . -name "*.pdf" -type f | wc -l

Extract text from PDF

Use poppler-utils package command pdftotext (which is probably already installed on linux, can be installed with Xpdf tools on windows):

pdftotext filename outputname

for f in *.pdf; do pdftotext "$f"; done

Add something to each line

Use sed:

sed 's/^/stuff in front/; s/$/stuff at end/' "input.txt" > output.txt

In batch:

for f in *.txt; do sed 's/^/front/; s/$/back/' "$f" > output/"$f"; done

See Sed: an introduction for extensive tips.

Get list of files meeting criteria

I often have a folder of text files and need a list of which one contain some specific string. Easy using grep with the -l flag for listing filename only (not the matching lines):

grep -l "string" * > list.txt

Or the opposite, files without the phrase, using -L for filenames with no match:

grep -L "string" * > list.txt

Compare two text files

To compare two text files you can use diff or comm. I often use this to find the difference between two folders or lists of files (ls > filelist.txt).

Diff will show a report something like a git diff:

diff list1.txt list2.txt

Comm will output a list comparing the lines in the two files, grouping them by lines only in file 1, only in file 2, and in both. Add the options -1, -2, or -3 to not view one of those groupings. Generally, I just want the differences, so use -3 to NOT see the lines that are the same in both.

comm -3 list1.txt list2.txt > comparison.txt

Copy a list of files

I often have a folder of binary files, like images, where I need to sort out a small group to upload or move somewhere else. Often the list of files that need to move comes from some other source, such as a spreadsheet. To do it as a batch, I export a list of the filenames that need to move, one filename per line (just like if you did ls > list.txt), then use a bash loop like this:

for f in $(cat list.txt); do cp $f ../upload/; done

In this example, we use cat list.txt to print out the file contents to provide the variables for the loop to iterate over–this can be handy for a lot of pragmatic solutions!

Don’t store in history

If you add a space at the beginning of the line, then the command won’t be recorded in your history. This is important if you are manually adding some passwords or secrets in the command.

Use command output in another command

Sometimes you can’t pipe output into a command, you need the output inside the command instead (i.e. “command substitution”). Use $(command) in place. E.g. JUPYTER=$(which jupyter) julia (gives julia-lang location of Jupyter Notebook).

Handy Ref

Golden Flippy Things

Tue, 23 Oct 2018 00:00:00 +0000

Rules for the classic beer game:

each player must create their own golden flippy things (i.e. by taking bottle caps and pounding them flat and smooth with a hammer)
each player should have a bunch
each player should probably have an equal amount
each round all players:
- put one flippy into the pot
- shake out all of their golden flippy things at once
- then count up your flippies that landed with gold side up:
  - person with most gold up wins,
  - unless a player has all non-gold side up, then they win,
  - unless there is a tie, then all players have to put one extra flippy into the pot and no one wins this round.
- the winner collects all flippies in the pot

Mechanics make it so that once someone starts winning, they will likely keep winning, yet it is quite hard to finish players off.

these official rules date back to Pittsburgh, 2000.

Fake a photosphere with ExifTool

Fri, 21 Sep 2018 00:00:00 +0000

I like to incorrectly use photo features to see what interesting stuff happens. In this case, I want to upload any random image to Flickr and make it display as a photosphere.

Photosphere’s are just regular flat JPG images, but the recorded image is a projection (probably “equirectangular”)–just like a map is a flat representation of a globe. Photosphere viewers know how to display the image to correctly unwarp the projection to make it appear like a 360 degree view. The viewer relies on some embedded metadata to tell it that the JPG could be viewed in this way. If you upload a valid photosphere to Flickr, the photo page will trigger a special in browser viewer so you can check out the 360 view.

The upshot: if we add the correct metadata to an image, we can trick Flickr or other viewers into displaying the image in 360, with odd and fascinating results.

As far as I can tell, Flickr requires four XMP tags in the GPano namespace (it is similar on other platforms), which in xml would look like:

<GPano:FullPanoHeightPixels>5000</GPano:FullPanoHeightPixels>
<GPano:FullPanoWidthPixels>10000</GPano:FullPanoWidthPixels>
<GPano:ProjectionType>equirectangular</GPano:ProjectionType>
<GPano:UsePanoramaViewer>True</GPano:UsePanoramaViewer>

ExifTool

To manipulate image metadata you need ExifTool. Truthfully, nothing else works reliably.

On Linux install ExifTool as the Perl library, e.g. sudo apt install libimage-exiftool-perl. On Windows download the standalone package.

To use ExifTool, we start with the command exiftool and the file name of an image. For example, the command exiftool test.jpg will print out all the current metadata embedded in test.jpg.

From there we add flags to the command to add or change the image’s metadata. For example, the command exiftool -all= test.jpg will delete all embedded metadata, i.e. replacing all values with nothing.

To figure out how to add the photosphere tags dig through the XMP Tags docs to find the correct keys. For example, to build up the command flag we want to add XMP metadata (-XMP), from GPano namespace (-GPano), in the tag ProjectionType (:ProjectionType), with the value of “equirectangular” (="equirectangular"). Thus to add/change that one metadata value, the full command would be: exiftool -XMP-GPano:ProjectionType="equirectangular" test.jpg. We can add multiple tags simply by adding more flags.

Fake a Photosphere

Here are the steps:

edit image to 2:1 ratio (for reference, Pixel 2 camera shoots real photospheres around 9000px width by 4500px height).
remove all existing image metdata: exiftool -all= test.jpg
add the new fake photosphere metadata (be sure to replace the pixel dimensions with the actual size of your image): exiftool -XMP-GPano:FullPanoHeightPixels="4500" -XMP-GPano:FullPanoWidthPixels="9000" -XMP-GPano:ProjectionType="equirectangular" -XMP-GPano:UsePanoramaViewer="True" test.jpg
upload to Flickr or other viewer

Some Other Viewer

Checkout my project sphere-view that uses Pannellum photosphere viewer, in a minimal Jekyll template on GitHub Pages. You don’t actually have to fake the metadata if you use this viewer…

MRIC: Adapting Technology for Teaching

Tue, 18 Sep 2018 00:00:00 +0000

Sean Butterfield and Evan Williamson, “Adapting Technology for Teaching: The Creation of an Open Educational Resource”, Malcolm M. Renfrew Interdisciplinary Colloquium, University of Idaho, 2018-09-18.

Abstract: Last year, Sean Butterfield accidentally wrote a textbook, and it was Evan Williamson’s fault. Together, they developed an Open Educational Resource (OER) for teaching the Lionel Hampton School of Music’s sophomore music theory sequence – a core class for all music majors that has a reputation for being difficult and inaccessible. This OER not only made the class accessible financially, but it also accommodated an entirely new curriculum designed around an inquiry-based learning method. The online distribution system allows the class teaching assistants to update the textbook in real time, enabling students to take an active role not only in their own learning, but in designing a permanent resource that includes their own thoughts, wording and concepts. The presenters will give an overview of the course and discuss the software and distribution platforms. They also will explain how this pedagogy has the potential to allow any course to engage instructors and students in a more meaningful, effective and memorable way.

IntMus Links

Integrated Musicianship: Theory
IntMus Theory source code on GitHub

Think Open Fellowship (University of Idaho Library)
Open Definition (Open Knowledge International)
Open Music Theory, by Kris Shaffer, Bryn Hughes, and Brian Moseley, 2014+ (OMT source on GitHub)

Static Web tools:
- GitHub (cloud based platform for version control, collaboration, project management, and code sharing)
- GitHub Pages (free web hosting service)
- Jekyll (static site generator, provides the theme / templates)
- Markdown (simple plain text content)
Interactive features:
- abcjs (live editable music examples in the text, e.g. Happy Birthday in G major)
- Disqus (discussion around assignments, e.g. Unit 4 assignment)

Pandoc for PDFs

Wed, 05 Sep 2018 00:00:00 +0000

Pandoc, the “universal document converter”, is a command line utility to convert between countless file types. I use it in my writing workflow to convert from Markdown to PDF.

Check the docs for full installation instructions, but basically you need to install Pandoc and a LaTeX distribution. Pandoc converts your Markdown and your Tex / LaTeX distribution does the typesetting to create PDFs. Here is everything necessary on Ubuntu Linux:

sudo apt install pandoc texlive texlive-fonts-extra texlive-xetex

If you install MiKTeX on Windows (your LaTeX distro) plus Pandoc, most of these commands should work–MiKTeX’s package manager will pop up asking to install what ever is necessary.

Note: this was written on Linux using Pandoc 1.19.2.1, it doesn’t work quite the same on Pandoc 2+ on Windows. I will update when I get a chance…

To create a basic PDF, use the -o output flag and a file name with .pdf extension. For example:

pandoc -o output.pdf input.md

For writing your content Pandoc supports an extended version of markdown syntax with many enhancements such as footnotes and tables (similar to GitHub Flavor Markdown). If you include a YAML metadata block at the top of your file, the standard fields will be translated into LaTeX elements (title, date, author, etc).

To tweak your PDF layout / typesetting, you can pass LaTeX variables to Pandoc using the -V variable flag (see docs for all LaTeX variable options). If you know LaTeX, you can get fancy right away and use existing templates. However, you can also use a few very simple options to spruce up the defaults. For example, the default margin is very large, so you might want to use:

pandoc -V geometry=margin=1.25in -o output.pdf input.md

This will use the LaTeX package geometry to set all the margins to 1.25 inch. You could set each margin individually with geometry options margin-left, margin-right, margin-top, or margin-bottom.

To change the font I usually use the XeLaTeX engine, since it can use any normal font installed on your system. First, you need to figure out the correct names for installed fonts. Use this command:

fc-list : family | cut -f1 -d"," | sort

The full name listed will be passed as a variable. Alternatively, open LibreOffice Writer and look through the font names. However, the variable is picky about the exact name used, so check against the official name as given in fc-list. The command looks like:

pandoc --pdf-engine=xelatex -V geometry=margin=1.25in -V mainfont="Noto Sans" -o output.pdf input.md

Note: older versions of Pandoc use --latex-engine instead of --pdf-engine, which is likely if you installed Pandoc from a distro repository.

You can also use the pdfLaTeX engine, which requires specifically prepared LaTeX fonts (usually stored with a .sty extension). Preparing and using the fonts requires a bit of LaTeX savvy (in theory you can use lots of cool fonts, FontCatalogue). However, the texlive-fonts-extra package contains a few ready to go options. The command looks like:

pandoc --latex-engine=pdflatex -V fontfamily="electrum" -V geometry=margin=1.25in -o output.pdf input.md

Font size can be controlled using -V fontsize=12pt, however, the default template only supports sizes 10, 11, or 12. More sizes are supported using the extsize package (8pt, 9pt, 10pt, 11pt, 12pt, 14pt, 17pt, 20pt) which can be used by adding -V documentclass=extarticle plus the desired fontsize (if extsize is not installed, your LaTeX distribution should automatically fetch it).

Now we are getting a lot of commandline arguments to keep track of… You can simplify by declaring the LaTeX options as YAML metadata at the top of the document instead. You will still need to use the --pdf-engine=xelatex option, but won’t need to list out all the -V options. For example:

---
geometry: margin=1in
documentclass: extarticle
mainfont: Noto Sans
fontsize: 14pt
colorlinks: true
---

For my basic purposes, that is plenty of control, but of course you can go much further by learning more about LaTeX. Check danstoner for samples of PDF creation, Practical Typography for system font suggestions, and the Pandoc User Guide for more info.

GNOME on Ubuntu

Sun, 02 Sep 2018 00:00:00 +0000

Starting with 17.10+ Ubuntu ships with GNOME desktop rather than Unity. Here are some notes to start customizing:

Start Tweaking

When you first install, there are a few things to do right away to get Gnome on Ubuntu looking better. First, remove the junk from the dock, i.e. Amazon, by right clicking and selecting “remove from favorites” (to uninstall Amazon, open Ubuntu Software and look on the Installed tab). Add good stuff to the dock as you open it, by right clicking the dock icon and selecting “add to favorites”.

Then head to Settings to get the desktop looking better.

Settings > Dock, make the icons smaller (the default seems to take up way too much space). This is the configurations for Ubuntu’s customized version of the popular extension Dash to Dock.
Settings > Background to get your wallpaper set.
Settings > Devices > Display to set up your monitor and turn on Night Light if you want a red shift in the evening.
Settings > Bluetooth, turn bluetooth off if you don’t use it, since it is a security risk.

Next, install Tweak tool, gnome-tweak-tool, to give more complete customization options.

Finally, if you don’t like Ubuntu’s dock, you can remove it to go to a more vanilla Gnome experience.

Change Themes

The default style for 18.04 is the pretty ugly and dated “Ambiance”, the familiar default from Unity. Luckily, changing themes is incredibly easy with Tweak tool, a simple drop down in Appearance > Themes, plus a separate drop down for your icon set. There is a whole universe of themes out there, but to get started you can install a few decent options from the main repository:

sudo apt install arc-theme numix-gtk-theme numix-blue-gtk-theme materia-gtk-theme numix-icon-theme papirus-icon-theme moka-icon-theme faba-icon-theme elementary-icon-theme pocillo-icon-theme

Other themes can be installed via PPA or manually. I like Pop theme (see instructions in the middle of the readme).

Ubuntu 18.10+ ships with a new theme Yaru (formerly known as “communitheme”). You can also try it out in 18.04 by installing it as a snap app, snap install communitheme, then choose it as a session when you log in (click the little cog next to the password box).

Get Extensions

The cool part of GNOME is the flexibility of using shell extensions to quickly add functionality to the desktop. These range from new themes to menus to info widgets.

First, install the Extensions browser plugin for Firefox or Chromium. Then, install the helper sudo apt install chrome-gnome-shell on your system (works for all browsers). Now, you can install, manage, and configure everything from your browser by visiting the GNOME Extensions page. Configuration can also be done in Tweak tool > Extensions.

For example, add useful utilities:

Caffeine, disable screen lock to watch movies–I consider this an essential, and the app version used in other Ubuntu versions will not work with Gnome.
OpenWeather
Clipboard Indicator, saves history of clipboard.
NetSpeed, displays internet connection stats in top bar.

Or completely change the desktop functionality:

Dash to Dock, a customized version of this extension ships with Ubuntu to create the Unity-like dock on the left side of the desktop–you can uninstall the Ubuntu version and add the vanilla extension if you want.
Dash to Panel, puts dash into the top/bottom bar, sort of like windows 10.
Arc Menu, replaces “show applications” with a more traditional drop down menu organized into categories. Similarly, Gno-Menu adds a “menu” option to the top bar with a comprehensive apps menu.
gTile, add a button to resize windows based on a grid, something like a basic tiling window manager (that you manually adjust).

Organize Apps

The main application launcher has an icon for every installed app. To help organize this list, you can group items together using GNOME/Ubuntu Software, it’s just not very obvious how to do it (in fact people often think it is impossible)…

Open Software, click on Installed tab, then on the check mark at the upper right corner. Now click the box next to each app you want to add to a folder. Once the group is checked off, click the “Add to Folder” button at the bottom left. In the pop up, click on the folder you want to add them too, or click the “+” to create a new folder. Finally, click “Add” to finish creating the new group.

This process can be simplified using the Appfolders Management extension, which adds folder options when you right click on the application menu.

New Apps

18.04 contains some new apps not available in the older version repositories. Here are a few I like:

gnome-games (collection of games)
gnome-games-app (browse and organize games)
tilix (tiling terminal)
gnome-boxes (browse virtual boxes)

Keyboard shortcuts

F11, toggle full screen for any app
Super (i.e. Windows key between Ctrl and Alt), toggle “Activities” overview / switcher (or Alt+F1)
Super+A, show application menu
Ctrl+Alt + Up/Down arrow, switch between workspaces (or Super+Page Up/Down)
Alt+Tab, switch between windows (or Super+Tab)
Alt+F2, command window, type r to restart GNOME to recover from freezes
To sleep rather than shutdown, open the topbar menu, hold Alt and the shutdown power icon will turn into a pause icon.
You can also customize your shortcuts using Settings > Keyboard > Shortcuts.

Issues with snap apps and themes

A few of the standard applications that ship with 18.04 are now snap apps, rather than traditionally installed software. There seems to be a bug with them when you are using a non-default theme, and they either won’t open or just look unusably terrible. This is really frustrating. So uninstall the snaps and reinstall the traditional app.

sudo snap remove gnome-system-monitor gnome-calculator

sudo apt install gnome-system-monitor gnome-calculator

Wayland Issues

While Ubuntu 17.10 shipped with next gen display server Wayland as default, 18.04 switched back to the venerable Xorg. At the login screen, you can choose which type of session to start When you start up your computer and arrive at the login screen, click on the gear icon next to the “Sign In” button. This will display two options, one saying “on Xorg” or “on Wayland”.

Using Wayland introduces a few complications with some traditional applications, most notably with graphical apps that require sudo access (gparted, synaptic, gufw) and screen recorders. These issues are part of the reason 18.04 went back to Xorg.

The other option is to avoid using those problematic graphical apps. For example, gufw is commonly used to set up a simple firewall on your personal computer, but will not run on Wayland. Just go commandline and use ufw with out the graphical interface (the defaults are good for most personal uses).

sudo apt install ufw
sudo ufw status
sudo ufw enable

More stuff

Other than the complications mentioned above, most of the essentials to get set up are in my older Ubuntu tips.

If you want to do some more serious customization, GTK+ can be styled with CSS (docs). You can start a program with a debug window to get all the information, sort of like using inspect dev tools on a web browser:

GTK_DEBUG=interactive gedit

Older notes

If you have issues with your keyboard or touch pad, there is a bug with some laptops in the kernel shipped with 17.10–see my note on adding new kernels to Ubuntu.

Git Branch Notes

Fri, 01 Jun 2018 00:00:00 +0000

Branching is a Git fundamental that allows you to test out ideas in parallel to your main repository without disrupting the main/master copy. Notes about confusing things:

Remote branches

It is a bit confusing that branches are only local by default, e.g. when you git checkout -b example on your laptop, the new example branch is only on your machine and will not automatically be pushed to GitHub. Likewise, a new branch on GitHub will not automatically appear on your local machine when you use git branch to list branches.

A branch that is connected to a remote repository (GitHub) is called a “tracking branch”. When you git clone, the main/master branch is automatically created as a “tracking branch” with the GitHub repository as the “upstream branch”. However, git knows about the other remote branches, but does not automatically create them locally. Furthermore, after you create a new branch locally, and you want to push it to GitHub, you will need to add a remote.

To get information about all your remotes, git remote show origin lists everything tracked locally and how it lines up with your remotes.

Use a remote “tracking branch” locally:

Clone the repo, git clone https://github.com/user/repository.git
Checking git branch will display only the local branches, i.e. main/master if you just cloned it.
Check git branch -a to see all branches, including remotes, i.e. git knows about the other branches, it just didn’t create a local version of it yet. The remote branches will be named like remotes/origin/branch-name.
To create a local version of a remote branch, use git checkout branch-name (when there is no local branch with that name and it exactly matches a remote branch, git will automatically create a local “tracking branch”). This is the short cut to git checkout -b branch-name origin/branch-name. Now with the branch there locally, you can push and fetch/pull as normal.

Create a branch locally, then add to remote:

Create and switch to a new branch, git checkout -b branch-name. (reminder: this is a short cut combo of two commands, git branch branch-name and git checkout branch-name)
To add the branch to the remote, git push origin branch-name will create the remote branch and set the “upstream” for the local version. (this is a shortcut to git branch --set-upstream-to origin branch-name plus git push)

Update branch from main/master

In most workflows you might not actually want to do this to avoid weird histories, but…

Update your main/master branch, with git pull.
Switch to the feature branch that needs updating, git checkout feature-branch.
Merge main/master into the feature: git merge main.

Web intro

Sun, 20 May 2018 00:00:00 +0000

Idealistic Canadian video: How Does the Internet Work ?

A network is two or more computers connected. URLs are stand ins for Internet Protocol addresses (IP), which are numerical identifiers for devices connected to a network.

The internet is network of networks that connects devices.

Demo IP addresses:

Find your IP
Check another IP: ping www.google.com or nslookup www.uidaho.edu
Follow your traffic: traceroute / tracert / tracepath tracepath www.google.com or mtr --report google.com
Make a request: curl www.google.com

For an in-depth, but fun intro, checkout Julia Evan’s Networking Zine.

What’s in a URL

https://example.com/about?key=value#anchor

protocol :// domain . top-level domain (optional port :80) / path and filename ? query with parameters # fragment or anchor

Protocol: there is actually more than 100 defined Internet Protocols, however with URLs the most common are Hypertext Transfer Protocol (HTTP) or File Transfer Protocol (FTP).
Domain: the Domain Name System (DNS) is often described as a “phone book” that translates easy to understand web addresses into the appropriate IP addresses of servers with the desired information. A subdomain can be added in front of the main domain. For example, in lib.uidaho.edu > lib is a subdomain of uidaho, which is a subdomain of the top-level domain edu. Occasionally, you may see a port number at the end of the domain which refer to specific communication endpoints that the server is listening on. For the web the standard ports are :80 (http) and :443 (https).
Path and filename: these can be imagined like folders and files on the server (and actually are in static web sites). For example, /about/index.html, is the index file in the “about” folder. If no filename is given, by default the server provides the file named index.
Query string: is added to the end of an URL following a ? and is given in key value pairs. Multiple pairs can be separated by &. The characters must be URL encoded / escaped, since some characters such as space are not allowed in URLs. The queries are processed by the server or javascript, so what exactly they do depends on the page. For example, /index.php?title=Query_string&action=edit might return a wiki article in edit mode.
Anchor: are added to the end of a URL following a #. They traditionally correspond to specific id on an html page, but are often used by javascript to add other functionality. For example, /index.html#Chapter1 might jump to a chapter heading in the web page.

Data Visualization Resources

Sun, 06 May 2018 00:00:00 +0000

Resources from “Data Viz 101: Concepts and Tools” workshop 2015-10-14

Background

Charles Joseph Minard’s Map (1869), from wikimedia. Edward Tufte says it’s “probably the best statistical graphic ever drawn”.

Different areas:

Data Visualization
Information Visualization
Visual Analytics

Books:

Matthew Ward, Georges G. Grinstein, and Daniel Keim. Interactive data visualization : foundations, techniques, and applications, Second edition (Boca Raton : CRC Press, 2015).
Colin Ware, Visual thinking for design (Burlington, MA : Morgan Kaufmann, 2008).
Colin Ware, Information Visualization Perception for Design, 3rd ed (Burlington : Elsevier Science, 2012).

Visual Analytics

“Visualization allows people to offload cognition to the perceptual system, using carefully designed images as a form of external memory. The human visual system is a very high-bandwidth channel to the brain, with a significant amount of processing occurring in parallel and at the pre-conscious level. We can thus use external images as a substitute for keeping track of things inside our own heads.”

Tamara Munzner, “Visualization,” in Fundamentals of Computer Graphics (3rd edition), ed. Peter Shirley, Michael Ashikhmin, and Steve Marschner (Natick, MA: A K Peters, 2009).

“Visual analytics solutions provide technology that combines the strengths of human and electronic data processing. Visualization becomes the medium of a semi-automated analytical process, where humans and machines cooperate using their respective distinct capabilities for the most effective results.”

Daniel Keim, Gennady Andrienko, Jean-Daniel Fekete, Carsten Görg, Jörn Kohlhammer, and Guy Melan, “Visual Analytics: Definition, Process, and Challenges”, in Information Visualization - Human-Centered Issues and Perspectives, LNCS, ed. Andreas Kerren, et al. (Springer, 2008), 154-175. Available at http://hal-lirmm.ccsd.cnrs.fr/lirmm-00272779/document

Shneiderman

Visual Information Seeking Mantra:

Overview first, zoom and filter, then details-on-demand.

Type by Task Taxonomy (TTT)

Seven data types: 1-, 2-, 3-dimensional data, temporal and multi-dimensional data, and tree and network data
Seven tasks: overview, zoom, filter, details-on-demand, relate, history, extract

Ben Shneiderman, “The Eyes Have It: A Task by Data Type Taxonomy for Information Visualizations”, Proceedings of the 1996 IEEE Symposium on Visual Languages (1996): 336. Available at http://www.interactiondesign.us/courses/2011_AD690/PDFs/Shneiderman_1996.pdf

Preattentive Features

Visualizations and Viz tools need to be designed with human visual abilities in mind. For example, check out this research about “Preattentive Features and Tasks” (on youtube):

Christopher G. Healey, “Perception in Visualization”, http://www.csc.ncsu.edu/faculty/healey/PP/

Visualization Examples

The best way to get ideas of good ways to visualize data is to look at lots of examples. Try some of these catalogs:

Negative Examples:

Viz Resources

Flowingdata
Infovis Wiki
visualising data
Gapminder
Viz-Palette (helps you choose accessible color palettes)
The Pudding (visual essays)

Simple Web Based Tools

Google Charts
Raw Graphs (based on D3.js)
Data Wrapper
Charted
Vega Voyager (open alternative to Tableau in development)
Highcharts Cloud (free web editor version of popular commercial js library)
Flourish (free version with public data)

Libraries

HTML+JS: D3 (Alternatively, checkout C3.js, dygraphs, TauCharts, chartjs, or Vega)
Python: matplotlib (you will probably want to use it along with Pandas to manipulate the data. Alternatively, checkout Bokeh, seaborn, plotly, or ggplot)
R: ggplot (usually used in conjunction with other Tidyverse packages. Alternatively, checkout ggvis, Shiny)

Code Notebooks

Code notebooks such as Jupyter are becoming an essential tool in data science. They combine code blocks with text and visualizations into a shareable document. Many notebooks now have cloud hosted versions to lower barriers to getting started and promote reproducibility:

Try Jupyter, or share Jupyter notebooks with nbviewer or binder
Colaboratory (Jupyter environment hosted by Google designed to run TensorFlow without installing anything, with your notebooks stored in Google Drive)
Observable (JavaScript based data visualization for the web)
Code Ocean (a variety of notebooks and IDE available)
COCALC (was SageMath Cloud, now has Jupyter Notebook with lots of kernels, a LaTeX editor, and more)
Iodide and Pyodide (web focused notebooks implemented in browser, in alpha development)
Stencila (word processor + spreadsheet + code)
Azure Notebooks

Tableau

I rarely recommend non-opensource tools, but Tableau is fairly unique tool for visually exploring data in a flexible, powerful sandbox.
It is commonly used in enterprise settings, however, they provide free licenses for academic learning use. If your data can be shared publicly, Tableau Public is a good option. Also, checkout their Iron Viz competition for interesting examples.

Academic program:

Free for students and teachers
Not free for administration, see higher-ed solutions

Tableau Public

Tutorials:

Fluffy Pancake Recipe

Sun, 08 Apr 2018 00:00:00 +0000

Usually I like custardy pannaukakku, but occasionally I have a hankering for a very fluffy pancake. Here is my most successful recipe.

In a big bowl, mix up:

2 eggs
1 3/4 cp milk
2 tbsp sugar
pinch of salt
1 tsp vanilla
1 tsp cardamom

Take a whisk and beat it up really well. Then add:

1/4 cp butter, melted (gently melt in microwave, and let cool a bit so it is not hot)

and stir up. Finally, add:

2 cp flour
4 tsp baking powder

and very gently stir it in with the whisk–don’t mix it too much! Should be thick and lumpy.

Then let the batter sit for awhile, like 10 minutes. It needs to rest.

Heat up a pan with bit of cooking oil, and cook as normal pancakes (i.e. drop blob of batter on to pan, cook until gets bubbly, gently flip over and cook until golden).

Hot Fudge Pudding Cake Recipe

Sun, 08 Apr 2018 00:00:00 +0000

Inspired by memories of cook book for kids from when I was a kid, delicious, chocolaty, and very simple.

Heat oven to 350 degrees. Grease a square baking pan (or small oval casserole, or double recipe for big rectangle baking pan).

In a mixing bowl, add:

1 cup flour
2/3 cup sugar
1/4 cup baking cocoa
2 tsp baking powder
1/4 tsp salt
1 ts vanilla extract
1/4 cup vegetable oil
1/2 cup milk

Mix it all up into thick batter. Spread into the greased square pan. Use a spatula to scrape out the mixing bowl pretty clean and dry.

In the fairly clean mixing bowl, add:

1/4 cup baking cocoa
3/4 cup brown sugar

Mix it up. Sprinkle over the top of the batter in the pan (don’t mix in at all).

Prep (in electric kettle or microwave or whatever):

1 1/4 cup hot water

Pour the hot water over the pan (don’t stir or mix or anything).

Put into oven and bake about 40 - 45 minutes. The top will become a brownie like crust, with gooey hot fudge sauce below.

Remove from oven, let cool a few minutes, then serve immediately in bowls.

I like to serve with some milk or ice cream.

Pannukakku Recipe

Sat, 07 Apr 2018 00:00:00 +0000

Inspired by memories of Chocolate Moose in Ely, MN, and ancient ancestral foods, we wanted to create thick, custardy Pannukakku (a Finnish oven pancake). Partly based on a “Scandinavian food expert” website, we came up with this recipe.

Basic concept is to create a batter following the ratio of 1 egg : 1/4 cup flour : 1/4 cup milk, beating it very well. Add a bit of vanilla, sugar, baking powder, and lemon zest, then bake in a hot oven.

We are not very exact about the measurements, but here is an approximate recipe for a rectangle cake pan. We like to use enough eggs to create a fairly thick pancake, which will be more custardy in the middle.

In a big bowl, add:

8 eggs
2 cup flour
2 cup milk
2 Tbs sugar
2 tsp vanilla extract
1 tsp cardamom
dash of salt

Beat it up really well (I think you want to get some air into the batter), then let it sit for awhile. Preheat the oven to 400 degrees (or higher). After the batter has rested a bit, add:

1 tsp baking powder
a bit of lemon zest (or lemon juice)

And mix up well.

Take a pan (oven safe cast iron or glass cake pan), add a couple big dollops of butter, and put into the oven until the butter is melted and bubbling near browning. Pull out the pan, pour the batter in, and pop it back into the oven.

Bake for around 15 minutes, until golden on top (it will puff up a lot!).

Meanwhile, whip up some fresh whipped cream.

Serve chucks of the pannukakku topped with whipped cream and fresh or canned berries. (Alternatively, serve with syrup or with lemon and butter)

Open Image Editing Workshop

Wed, 04 Apr 2018 00:00:00 +0000

MILL workshop 2018: Learn about free and open-source applications that make advanced media editing possible without expensive software! This session will introduce the most popular open-source media editing applications and get you setup to use GIMP for photo editing.

When thinking about editing images and other media, a certain super expensive, close-source ecosystem usually comes to mind (i.e. Adobe)–to the point it has even become a verb, “photoshopping”. However, there are plenty of excellent open source alternatives that can power your media editing workflows for free. These cross-platform, open source options respect your rights as a user, and you won’t be locked into a subscription service or platform.

Please note: we often mention open source software as an “alternative to” some proprietary / commercial option. However, none of these open source projects are designed to specifically mimic or replace a proprietary tool. They are developed and guided by their own community and goals.

These notes are a sketch outline with links to lots of free tools. For open image editing tutorials and resources, the go to resource is PIXLS.

GNU Image Manipulation Program (GIMP)

When we talk about photo editing, we usually mean “raster images” or “pixel editing”. While “photoshopping” is a verb, GIMP is a powerful and popular free and open alternative.

Once installed I suggest a quick tweak: click “Windows” > “Single window mode” for a more usable experience, unless you are comfortable with the floating docks. If you have some favorite keyboard shortcuts, check in “Edit” > “Preferences” > “Interface” to configure them. For help, click “Help” > “User Manual”.

Download a couple free images to practice on:

Unsplash.
Flickr Commons.

The basic GIMP interface:

Left dock: tool box and tool options.
Center: image window.
Right dock: layers, history, channels; brushes options.

Demo outline:

Load image (File > Open).
Adjust view on image window (zoom, quick move in corner, F11 for full screen).
Image > Transform > rotate.
Select tool, rectangle select, adjust selection.
Image > Crop to selection.
Filters > Enhance > Unsharp mask.
Colors > Levels.
Image > Scale image.
File > Export as (add file extension to control output. Note: “Save as” is only .xcf format, similar to Photoshop’s .psd, which stores the full image and editing history so it can be worked on losslessly in the future. However, it can not be opened by other software and is much bigger!)
History tab, step back.
Try some tools: Airbrush, Smudge, Text, Color Picker.
Select, Cut; Select, Copy, Move.
Foreground / Background color.
Export review!
Load another image review!

Note: GIMP used to have a great plugin ecosystem. However, it is temporarily suspended due to security concerns.

Raw processing workflow

Advanced collection management and processing of raw image file types, including metadata editing, color management, batch processing, and automation (something like Adobe Lightroom).

darktable, full workflow.
RawTherapee, focus on raw processing.
digiKam, focus on management.

Other Raster image tools

Krita, community focused on “digital painting” application.
Luminance HDR, HDR utility.
Hugin, Panorama stitching utility.
Film Emulator, web based tool to emulate to color of old film types with your image.

Advanced:

ImageMagick, powerful image manipulation tool, used via CLI or by many other applications (my get started notes).
G’MIC, CLI, library, and GIMP plugin for advanced image processing.
ExifTool, image metadata utility, the only reliable way to read and write photo file embedded metadata. Or you can try Exiv2.
ImageJ, powerful batch image processor designed for scientific analysis (or my notes for less scientific purposes).

Vector editing

Vector editing programs (something like Adobe Illustrator) use standards such as SVG to record information about the positions, paths, and properties of the components that make up a graphic, rather than the individual pixels seen in raster images. This means the image can be scaled to any size with out loss of detail.

Inkscape, powerful, with more advanced features than AI under the hood, but a steep learning curve. Luckily, there are a lot of tutorials, check the official list, “Mastering Inkscape in 2018”, or “Awesome new Inkscape features”. The developers collaborate directly with the SVG standards committee to innovate the open specification.
Not open source, but cost-free, easy to use: Vectr or Gravit Designer.
Traditional, but handy: Draw in LibreOffice or Karbon in Calligra.
Super basic, but quick and open: SVG-edit.

Video

Unfortunately, none of the open video editors are great just yet, and all seem a bit unstable at times. I have used OpenShot successfully on many projects, fairly easy to use. Others say Kdenlive is stable and professional (check out the “Current state of Linux video editing” for some in-depth reviews).

Decent: OpenShot and kdenlive.
Less stable in my experience: Shotcut, Pitivi, and avidemux.
Pro, not open, but cost-free: Lightworks (full editing software used by professional studios, with export limited to 720p) or DaVinci Resolve.

Others

Blender, super powerful 3D modeling, animation, rendering, and video. Complex, professional grade tool. The developers host movie projects that produce completely free and open short films.
Audacity, audio recording and editing.
Scribus, desktop publishing.
vidcutter, handy utility for cutting video files.

R and RStudio on Linux

Tue, 03 Apr 2018 00:00:00 +0000

Note: this is out of date and due to new licensing terms, I no longer suggest using Anaconda.

Installing RStudio is a surprisingly complex on Linux, or at least much more work than expected. However, there is one easy way if you use Anaconda Python distribution. This note describes both methods.

RStudio and R with Conda

Anaconda is a major Python distribution that comes packaged with a huge number of scientific Python tools, including Jupyter Notebook and several difficult to install SciPy packages. Futhermore, it provides the conda package and environment management tool that simplifies installing, updating, and managing everything. Overall, it is very handy and powerful, simplifying the initial set up of a data science / SciPy environment.

Conda can also install and manage R on your system, which is nice since it makes keeping everything up-to-date easier.

Download Anaconda.
Install using the downloaded script (full instructions), bash ~/Downloads/Anaconda3-5.1.0-Linux-x86_64.sh.
Install R and RStudio: conda install -c r r-essentials rstudio.

You can now start RStudio by typing rstudio in the terminal (it will not show up in your applications), or start an R prompt by typing R. As a bonus, r-essentials automatically installs the IRKernel is that can be used in Jupyter Notebook. When you start Jupyter (jupyter notebook), R is available in the options to create a new notebook.

Anaconda gives you the choice of using MRO or normal R. Use conda packages mro-base or r respectively.

Note: When installing on Fedora I have run into issues where some applications no longer work because they are trying to use Anaconda rather than the default system Python (I haven’t had this happen on Ubuntu). The installer adds a line to your .bashrc in order to make Anaconda available on your path. Open sudo nano ~/.bashrc, and look for a line like:

# added by Anaconda3 installer
export PATH="/home/username/anaconda3/bin:$PATH"

I am conflicted about Anaconda. It is undeniably useful and powerful, simplifying getting started with advanced Python and scientific computing. It is great for teaching, thus has been adopted by organizations such as Software Carpentry. However, as it grows, it is a for-profit enterprise company and building “partnerships” with huge tech companies. This has led to questionable activities, such as adding a prompt to install Microsoft Code Editor to the Anaconda installer. I am actually a Code user, but it seems like classic shady software marketing technique that makes me uncomfortable. The good part of these “partnerships” is sustainable funding and development from the biggest drivers of data science industry. However, it also reduces the community driven spirit and values of open source software.

Traditional Install

To get RStudio running first you need R. R is available in some distro repositories, but is often out-of-date. Specific information on the best way to install can be found from CRAN in the Readme files in the bin/linux directory (not exactly the most user-friendly way to provide information, but the readmes are comprehensive).

On Fedora, the repository version is good, just use the metapackage: sudo dnf install R.

On Ubuntu things are a bit more complicated:

Choose your closest mirror from the CRAN mirror list, for example https://cran.cnr.berkeley.edu/.
Open your apt sources: sudo nano /etc/apt/sources.list
At the bottom, on a new line add the CRAN source, following the pattern deb + <mirror url> + /bin/linux/ubuntu + <ubuntu version name>/, for example: deb https://cran.rstudio.com/bin/linux/ubuntu artful/, then save (Ctrl+X, then Y).
Add the secure key from CRAN, sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E084DAB9
Update sudo apt update
Install R: sudo apt install r-base r-base-dev r-recommended
Install dependencies necessary to build R packages: sudo apt install libxml2-dev libssl-dev libcurl4-openssl-dev libopenblas-dev

Next, install RStudio:

Download the package from RStudio.
Double click the package to install using the Software center.

R Resources

R for Data Science, Grolemund and Wickham.
Text Mining with R, Silge and Robinson.
SWC Programming with R or R for Reproducible Scientific Analysis.

Setup ImageMagick

Mon, 02 Apr 2018 00:00:00 +0000

ImageMagick is a powerful open source commandline utility for processing image files that is used under the hood by numerous other tools and websites. Once you get the hang of it, you can use it directly to do a lot of interesting things, from simple batch conversions to complex rendering.

Note: ImageMagick commands made a major change between version 6 and 7+. The docs on the main website refer to version 7+, where all commands start with magick. Previous versions used several different commands, such as convert, instead. A mirror site contains all documentation about “legacy” ImageMagick.

Install ImageMagick

Install ImageMagick and Ghostscript (which enables it to work with PDFs). Setup is easy on Linux, however, it will likely be a legacy version (see note above):

Ubuntu: sudo apt install imagemagick ghostscript.
Fedora: sudo dnf install ImageMagick ghostscript.

On Windows:

Download the ImageMagick installer.
Run the installer with default options.
Download the Ghostscript installer (you probably want the AGPL Windows 64 bit version).
Run the installer with default options.

Also on Windows, if you want to follow the commands in this note, be sure to have a bash terminal, such as Git Bash (packaged with Git installer) or Cmder.

Using ImageMagick

The full ImageMagick docs do a nice job of introducing commandline processing concepts. However, here is the basics.

Basic pattern: <command> <image options> <input filename(s)> <image operator(s)> <output filename(s)>

First, use a built in pattern to create a test image: magick -size 640x480 pattern:checkerboard checkerboard.png

Convert into a JPEG: magick checkerboard.png output.jpg

Resize: magick checkerboard.png -resize 75% out2.png

Crop: magick checkerboard.png -crop 120x120+10+5 test-cr.png

Combine all PNG in the folder into a PDF: magick *.png test.pdf

Combine all JPEG in the folder into a GIF: magick *.jpg test.gif

Create an image from first page of PDF: magick test.pdf[0] new.jpg

Resize all PNG in a folder, convert to JPEG, and sequentially number: magick *.png -resize 200x200 thumb%d.jpg

Resize all PNG in a folder, convert to JPEG, but keep original file name: magick *.png -resize 300x300 -set filename:f '%t' '%[filename:f]-thumb.jpg'

Check the commandline options page for more details.

Create PDF Thumbs

In this example, I want to create an image representing a PDF by grabbing the first page from each PDF in a directory and creating a JPEG. A few approaches:

use Magick’s batch abilities: magick *.pdf[0] -resize x500 -set filename:f '%t' '%[filename:f]-thumb.jpg'
use simple Magick command in a Bash loop: for f in *.pdf; do magick "$f"[0] -resize x500 -flatten "${f%.pdf}.jpg"; done

For higher quality result, add -density 600, for example: for f in *.pdf; do magick -density 600 "$f"[0] -resize x500 -flatten "${f%.pdf}.jpg"; done. Before working with the PDF, Magick must render it at a specific resolution. The default density is 72 dpi, so bumping it up to 300 or 600 will greatly enhance the quality of most image, but also significantly slow processing.

If you need higher speed processing the PDFs, it might be faster to use Ghostscript or Xpdf directly to create the images:

Ghostscript: for f in *.pdf; do gs -q -o "${f%.pdf}.jpg" -sDEVICE=jpeg -dLastPage=1 -r300 "$f"; done gets the images.
Xpdf tools provides the pdftopng utility that can quickly create PNG images from PDFs. For example, for f in *.pdf; do pdftopng -l 1 -r 300 "$f" "${f%.pdf}"; done, would grab the first page at 300 dpi.

Then use magick to resize the JPEGs, something like for f in *.jpg; do magick "$f" -thumbnail x800 -flatten "${f%.jpg}-sm.jpg"; done.

If ImageMagick seems too slow, you can try GraphicsMagick which implements all the same tools and options, but is more optimized for speed. In the example commands above, the only difference is magick would be replaced with gm convert. When working with images of very large pixel dimensions, gm seems more efficient than magick.

PDF Error on Ubuntu

If you are using the older version of ImageMagick installed from Ubuntu repository (v. 6.9.x, and thus using convert), you will probably get an error trying to create images from PDFs. Version 6 had a security policy against using Ghostscript functions due to an security issue in gs. However, the Ghostscript version in the repositories is more up to date than the ImageMagick version–so the issue probably doesn’t apply.

Check your Ghostscript version:

gs -v

If you have > 9.24, you can safely override ImageMagick’s policy. Open the policy file:

sudo nano /etc/ImageMagick-6/policy.xml

Find this section:

<!-- disable ghostscript format types -->
  <policy domain="coder" rights="none" pattern="PS" />
  <policy domain="coder" rights="none" pattern="PS2" />
  <policy domain="coder" rights="none" pattern="PS3" />
  <policy domain="coder" rights="none" pattern="EPS" />
  <policy domain="coder" rights="none" pattern="PDF" />
  <policy domain="coder" rights="none" pattern="XPS" />

Then comment it out:

<!-- disable ghostscript format types -->
<!--
  <policy domain="coder" rights="none" pattern="PS" />
  <policy domain="coder" rights="none" pattern="PS2" />
  <policy domain="coder" rights="none" pattern="PS3" />
  <policy domain="coder" rights="none" pattern="EPS" />
  <policy domain="coder" rights="none" pattern="PDF" />
  <policy domain="coder" rights="none" pattern="XPS" />
-->

And save.

Random helpful things

Sometimes TIFF files have compression inside, which some other utilities can’t read. You can pass them through imagemagick to resave uncompressed: magick compressed.tif out.tif

Or as a batch:

for f in *.tif; do magick "$f" -set filename:f '%t' 'magick-%[filename:f].tif'; done

Connect to AirVandalGold wifi on Linux

Sun, 01 Apr 2018 00:00:00 +0000

There is no automatic set up utility for University of Idaho’s AirVandalGold wifi network for Linux users. 2020 Update: CA certificate step is no longer needed which greatly simplifies the process and ITS finally added instructions (with screenshots from an very out of date Ubuntu desktop). However, VandalGuest’s new captive login portal does not seem to work.

I got it to work using these ITS instructions on Ubuntu and Fedora:

Open Settings > Wifi, then click on AirVandal Gold, which should pop up the full manual settings (alternatively, from the Wifi menu, choose ‘Edit Connections’, then click ‘Add’. Choose connection type ‘Wi-Fi’, click ‘Create’. Give it a connection name (this is for your own use), AirVandalGold. Under Wi-Fi tab add SSID: AirVandalGold.)
Under Wi-Fi Security tab, select/add these options:
- Security: “WPA & WPA2 Enterprise”
- Authentication: “Protected EAP (PEAP)”
- Anonymous identity: [leave blank]
- CA Certificate: check “No CA certificate is required”
- PEAP version: “Automatic”
- Inner authentication: “MSCHAPv2”
- Username: [your vandal web id]
- Password: [your vandal web password]

You should now be able to connect!

VPN

You can also connect to UI VPN on Linux:

Request VPN access by starting an ITS ticket if you aren’t already set up with permissions.
Download the Cisco AnyConnect VPN software from ITS software page.
Follow VPN install instructions. Even though it is out of date, the sh install script should work, taking just a few seconds.
Open Cisco AnyConnect, in the “Connect to:” box add vpn.uidaho.edu, then use your UI netid and password. Your Duo two factor id will be prompted to complete the log in. The first time you connect, the client will immediately download and install an update.

Once connected via VPN, it is like you are connected to the network on campus. You can also then connect to the shared drives (although these are being phased out for OneDrive) and remote desktop.

To connect to shared drives, you can use the “Other locations” > “connect to server” option in Files. However, this is usually very slow because Ubuntu uses an out-of-date SMB version to retrieve directory information (see note). It is better to manually mount the share, once you have connected via VPN.

Remote desktop works well with the default Remmina remote client:

The windows computer must be set up to enable remote desktop (must be admin).
Connect via VPN.
Open Remmina and click “create new connection profile”.
Use the computer name given during windows setup, something like xxxx.uidaho.edu (don’t fill in other fields).
accept security certificate.
use UI netid and password to log in (leave Domain blank).

Boot a Live USB

Mon, 26 Mar 2018 00:00:00 +0000

Burning a live USB is a great way to test out Linux distros (see Burn SD/USB and Linux Intro notes). However, for it to work you need to boot from USB. For most systems this is not the default and you will have to intervene as your computer starts up.

The boot process is controlled by the BIOS or UEFI installed by your computer manufacturer. Since each company creates a slightly different firmware, there is a lot of variation in how to exactly interrupt normal startup–but this note tries to gather some general tips together.

Backup

If you are messing around with your computer, its a good time to take some basic precautions to give you peace of mind–these are best practices that you should follow anyway! Backup of all your files on external storage to make sure you have everything you really want saved in multiple stable locations.

On Mac, review how to get to macOS Recovery. On Windows, make a Windows Recovery Drive on a USB stick (search your system for “recovery” and the option to create a recovery drive will pop up. This ensures you can reinstall or repair Windows if necessary).

Windows 10 / 8 Prep

Windows 10 / 8 overwrite parts of the file system to enable quicker boot times, bypassing the traditional boot process. This “Fast Startup” needs to be disabled or you will not have the option to boot from USB and your linux install will not be able to properly identify the Windows boot manager.

Open the Control Panel > Power Options.
Click “Choose what the power button does”, then “Change settings that are currently unavailable”
Uncheck the box next to “Turn on fast startup (Recommended)”
Completely shutdown your computer (not sleep, hibernate, or restart).

Secure Boot

Newer PCs have a Secure Boot feature that prevents unsigned OS from booting on the system. Major Linux distributions (Ubuntu, Fedora, SUSE) have secure signatures that allow them to run on systems with Secure Boot (you do NOT need to disable it). However, smaller distros do not, so the Secure Boot will have to be disabled. If you try the Boot USB tips below, but after selecting the USB the system skips it and boots into the normal OS–Secure Boot is probably at work and should be disabled.

To disable Secure Boot:

Enter your BIOS menu during bootup (see tips as below) or while in Windows: go to the shutdown menu, hold the shift key and click Restart, from the recovery menu select “Troubleshoot” then “Advanced Options: UEFI Firmware Settings”.
Look for the Secure Boot settings under the Security, Boot, or Authentication tab (each manufacturer is different).
Set Secure Boot to Disabled.
Save changes and exit

(See more advice from Microsoft help)

Boot USB

With your system fully shutdown, plug your Live USB into your computer.

If your computer currently has Linux or Mac installed, it might be easy:

On Linux: turn on the computer, when you see the GRUB boot menu, use arrow keys to select “System Setup”. This should open the machine’s BIOS/UEFI menu, which should have a boot device option.
On Mac: turn on the computer, immediately press and hold the Option key. This should bring you to the “Startup Manager” allowing you to select the USB drive (more tips on Mac help).

Otherwise, try the following tips:

Press the power button–then watch closely! Each device is different, but just as the computer starts to boot you should see a screen with the manufacture’s logo and a message that tells you which key to press to “interrupt the boot process”, enter BIOS/UEFI, or “choose a boot device”. Sometimes it flashes by so fast you can’t read it!

The key is usually F12, Esc, F11, F10, F9, F1, F2, or DEL (here are some tips from Pendrivelinux or gpost). If your laptop has a compact keyboard, be sure to use the correct key combo (often fn + key). Click the key repeatedly until you get a response, if it flashes by too quick and starts booting Windows, hold the power button to shut down, then try again.

Here are some possible outcomes:

Choose Boot Device: If you are lucky, your vender provides the “choose a boot device” option. Once you click the correct key, a small menu will appear listing the available disk drives in the system. Look for your USB’s label and size to identify the correct drive. Use the arrow keys to highlight the USB drive, then click enter to boot–Success!
BIOS Boot Order: For older systems if you get into a BIOS menu, but can’t find a “choose a boot device” option, look for a “Boot order” setting. Use the menu to change the boot order, moving USB drives to the top (traditionally, optical drives are at the top, then hard drives). Save and exit the BIOS menu and reboot–the USB should boot.
Locked down UEFI: On some Windows 10 / 8 machines, UEFI has been made a bit more complicated and there is no boot device or boot order options at start up. Allow Windows to boot up. First, make sure you have disabled “Fast Startup” as described above. Then, go to the Windows shutdown menu, hold the shift key and click Restart. This should bring up a blue boot options menu, select “Use a device”, then select your USB. Alternatively, try Settings > Update & security > Recovery, then look for “Advanced startup” and click “Restart now”.

There is a lot of variation in exactly how to access BIOS and UEFI options from different computer companies, so if none of this seems to work, try searching for help on your specific model and manufacturer.

Fedora Setup

Sat, 17 Mar 2018 00:00:00 +0000

After being a long-term Ubuntu user, while using 17.10 and looking ahead at 18.04, I suddenly lost a bit of hope for the distribution and decided to try some others for awhile (especially since I have been manually installing new kernels due to bugs with my laptop with 17.10).

First, Fedora. I choose Fedora because it is a bit more cutting edge than Ubuntu, yet still fairly stable (and running a newer kernel that works correctly with my laptop). I have found that although Debian/Ubuntu is known for gigantic repositories, Fedora plus RPM Fusion has more packages that I am interested in and they are more up-to-date. Fedora is also interesting because of its connection to the huge enterprize Linux company Red Hat. There are a lot of good things about Fedora philosophically, I am liking it but I haven’t been totally sold on it yet… anyway, here are my notes about getting started:

Installer

Fedora uses their “Anaconda” installer, which is not as immediately user friendly as Ubuntu’s. Unlike Ubuntu, it does not automatically detect existing operating systems and offer to dual-boot or other options. However, dual-booting is still possible, it is just not well documented and the UX is poor.

To dual-boot with an existing OS, chick on the install location icon and select your hard drive. In the options below, check “Automatic” and “I would like to make additional space available”. Then click “Done” at the top. This will open a window where you can choose existing partitions to delete–i.e. select your old Linux partition you want to over write. The selected partitions will be deleted, and Fedora will automatically set up new ones in the freed up space.

In my case, this seemed to work good. However, Grub does not detect Windows on my machine for some reason. I haven’t been able to fix this yet, because Grub uses os-prober and doesn’t find anything except Fedora.

This highlights the main frustration I have run into so far with Fedora: there just isn’t as much help online. If you search for any issue you will get a thousand Ubuntu answers, and maybe one incomplete/out-of-date Fedora one.

Try Ask Fedora…

Get started

Once Fedora is installed, I noticed start up time is longer than expected.

Update: do a sudo dnf upgrade --refresh, this will refresh the package list and update everything. This first one might be big and will require a restart to finish updating.
Tweaks: sudo dnf install gnome-tweaks, to get started customizing the look and feel. I turn off animations right away.
Enable RPM Fusion: follow the links on the Fusion Configuration page to set up the extended repository for Fedora (the main repos are purely Free software only).

Get Gnome Extensions

Fedora comes with a few GNOME extensions installed, head to Tweaks > Extensions to try turning them off and on. To get more, you have to add some things, because the current GNOME shell extensions browser plugin doesn’t work with up-to-date Firefox. You need Chromium and the host app to get started:

sudo dnf install chromium chrome-gnome-shell

Once installed, head to GNOME Extensions to try stuff out. Here are two I like:

Caffeine, disable screen lock to watch movies
Dash to Panel, puts dash into the top/bottom bar, sort of like windows 10

Get themes

The default Adwaita theme isn’t that lovely. There are a bunch of themes in the Fedora repos, so install a bunch and play around:

sudo dnf install arc-theme numix-gtk-theme numix-icon-theme numix-icon-theme-circle bluecurve-gnome-theme bluecurve-icon-theme gnome-shell-theme-selene bluecurve-metacity-theme nodoka-theme-gnome elementary-icon-theme pop-gtk-theme pop-icon-theme moka-icon-theme evopop-gtk-theme evopop-icon-theme arc-theme adapta-gtk-theme paper-icon-theme

Then open Tweaks > Appearance to switch between the Applications and Icons themes to see how they look.

Get more apps

Use Software, or search Fedora Packages and use sudo dnf install <packagename>.

Jekyll

My next set is always to get Jekyll set up. You could use the main repositories, like sudo dnf install ruby ruby-devel rubygem-jekyll, but you end up with a fairly out-of-date Jekyll.

Unfortunately, you need a bunch of stuff. This worked for me:

sudo dnf groupinstall "C Development Tools and Libraries"

sudo dnf install redhat-rpm-config

sudo dnf install ruby ruby-devel rubygems-devel

gem update --system note: vulnerabilities recently discovered in gem, update!

gem install jekyll bundler

Fonts

You might want a few more fonts, open Software, click Add-ons > Fonts. Scroll through and install a bunch. I also do: sudo dnf install google-noto-mono-fonts adobe-source-code-pro-fonts.

Also, you can download Fira Code, look in the ttf folder, then double click on each of the font weights. This opens up the font install window, click the Install button to add to the system.

Alternative Kernels on Ubuntu

Thu, 01 Feb 2018 00:00:00 +0000

I encountered an amazingly annoying keyboard and touchpad bug in early 2018 with the linux kernel that ships with Ubuntu 17.10. This led me to switch to a newer mainline kernel where the bug is fixed. At first it sounds complicated to re-install the most basic parts of your operating system, but in fact it is pretty easy!

Check Current Kernel Version

The command uname -r will list your current kernel version. If you want to see a list of all kernels currently installed, try dpkg --list | grep linux-image (yes, you probably have more than one!).

Boot with an Older Kernel

As you install regular updates, Ubuntu will add new versions of the kernel. Unless you sudo apt autoremove, these older versions will be available on your system in case you need to roll back. When you hit the GRUB menu during system start up, arrow down and select “Advanced options for Ubuntu”. This will give you a list of installed kernels. Simply select one of the older ones to boot up and see if the bug goes away.

Add New Kernels

To get Ubuntu packages for the new kernels, visit the Ubuntu kernel ppa. The kernel versions are numbered like v4.15.1 and fall into several categories (RC, Mainline, Stable, or Longterm). Mainline is the newest version line (e.g. 4.15), and is continuously updated with new Stable subversions (e.g. 4.15.1). The newest Mainline kernel has the most up-to-date fixes and performance improvements, but also has the most unknown bugs. Older Stable and Longterm kernels are regularly updated with patches from the Mainline, thus may have issues, but most are known bugs that have documented fixes. There is a trade off between the most up-to-date and more stable kernels (see “Which kernel version is stable” for a good discussion), thus most distros maintain a LTS version with their releases.

From the the Ubuntu kernel ppa, click on the newer version you want to install. Scroll down to the “build” section for your computer’s architecture, most likely “amd64” and “generic” for a 64-bit laptop. You’ll need three .deb files from the list: headers all, headers generic, and image generic (i.e. not the “lowlatency” ones). Download the three packages for the new kernel, for example, the version number is replaced with “…” in the list below:

linux-headers-... _all.deb
linux-headers-... -generic... _amd64.deb
linux-image-... -generic... _amd64.deb

Next, to simplify install, I create a new folder and add the three downloaded .deb packages. In the new folder containing only the three kernel packages, run sudo dpkg -i *.deb (it is best to install all at once, since they are dependencies of each other). Once the install has finished, restart and you will be using the new kernel. If you encounter an issue with the new kernel, rollback to the older version using the GRUB menu.

Keep in mind that the manually installed kernel will not be updated or automatically removed. Once your distro adds a newer kernel version, automatic updates will kick in again and install the newest version. Thus, if you want to keep on the regular updates, it is best not to jump a major version ahead if possible, since your distro may not ever catch up.

To remove manually installed kernels, it is easiest to use synaptic package manager (note: does not run in a Wayland session). Search for “linux-image”, mark the versions “for Complete Removal”, then click Apply. Just be sure not to remove ALL your kernels, or you will have nothing to boot!

Ubuntu essentials

Thu, 02 Nov 2017 00:00:00 +0000

Ubuntu is great, but here are some essentials and tips to get started and fix some issues that I regularly encounter. This post is from Ubuntu < 17.04, when Ubuntu shipped with the Unity desktop. Since 17.10+, Ubuntu ships with Gnome desktop, so some of this post is not relevant or out-of-date–check my notes about Gnome on Ubuntu for tips.

Install essentials

The quick and easiest method to install packages is to use the command line, sudo apt install packagename. Otherwise, search the Software Center.

Tweak Tool is handy to change the look & feel of the desktop, exposing many settings that are normally buried deep in config files. On Gnome (17.10+) gnome-tweak-tool or Unity (<17.04) unity-tweak-tool.
Play MP3s: many common media codec are not free and open source, requiring complicated licensing. Thus Ubuntu does not ship with them. There is an option to add them during install, but I don’t since it complicates the process. Install manually using ubuntu-restricted-extras (note there is a different package name for each desktop and version, so confirm by searching on packages).
Play videos: VLC, get a better video player right away, vlc.
Play DVDs: DVDs can’t be played until you install some non-free stuff. The codecs are not ready to go in the main repositories, so the easiest way seems to be to build it yourself: sudo apt install libdvd-pkg then sudo dpkg-reconfigure libdvd-pkg. You may also need to add these packages sudo apt install libdvdnav4 libdvdread4
Caffeine, turns off screen lock so that your screen doesn’t turn off while you are watching youtube! On Unity look for caffeine-indicator in the Software center. On Gnome, use the Caffeine extension.
Text editor: tweak the settings of Gedit, and add Visual Studio Code or Atom.
Git: sudo apt install git do it!
Simple firewall: gufw. Check out some more extreme security suggestions.
Image editing: GIMP, gimp (the first time you open it, click “Windows” > “Single window mode” for a more usable experience). If you use raw images, check out Darktable. To play with HDR, luminance-hdr.
Ebook manager: Calibre, if you have ebooks, this helps you manage them and connect with reading devices. It also has great built in editing and conversion tools.
KeePass2 (keepass2) or KeePassX for passwords.
build-essential adds Make and compilers necessary for building a lot of software.
curl is often to grab web stuff on the command line.
A few extra fonts: fonts-firacode fonts-noto
Firefox extensions: uBlock Origin, Facebook Container, Multi-account Containers, Side-view, Notes, OneTab
Disk utilities: Etcher for burning USBs, and GParted for formatting.

Languages

Python: Ubuntu comes with a system Python version installed. If you want to work with Python, its a good idea to install a more up-to-date version for your user account. I suggest Anaconda, a scientific Python distribution with comes packages with everything you need, including iPython and Jupyter Notebook. Then set up Jupyter with Py 2, 3, and R.
Java: default-jre or default-jdk (this tends to be up-to-date, some apps may require an old version, search packages for find them).
Ruby: probably best to use RVM (see note), although you can use the repository (ruby-full), but it is slightly out of date.
Processing3: download
Arduino: download
NodeJS: use nvm (this seems to be the best method; install nvm, then nvm install --lts), or official NodeSource ppa, or repository version nodejs (which maybe a bit behind)

Older suggestions for Ubuntu < 17.04 with Unity

Tweak settings

Open the Settings app:

Appearance > Behavior > set options to Show the menus in the window’s title bar and Always displayed.
Screen Display > set the Scale if everything looks tiny on your High DPI screen. Start with this setting before tweaking other scaling options.
Use Unity Tweak to customize everything else…

If you like the night time color shift, check out Redshift.

Mouse doesn’t seem quite right

If its a logitech mouse, install solaar with Unity/gnome extensions, sudo apt-get install solaar-gnome3. Then unplug the mouse and restart the system. Or try lomoco to manually configure.

gvfsd-smb-browse uses 100% of one CPU core

This is an odd bug that has effected people for a long time, but hasn’t been fixed, particularly on 16.04. If you notice your machine heating up, check you system monitor and stop gvsd-smb-browse process, don’t kill it. There is 100 fixes available online, but all are sort of sketchy and odd. The best option: install samaba, sudo apt-get install samba. It seems Ubuntu doesn’t have the full package, and once you install it, the bug seems to go away.

Gaming on Raspberry Pi

Mon, 18 Sep 2017 00:00:00 +0000

#MakeItMondays in the MILL: Gaming on Raspberry Pi!

Raspberry Pi is a wonderful little $35 linux computer–and it’s a lot of fun. Sure there is a lot of applications for physical computing or learning to code, but Pi also a handy open-source gaming machine! If you are interested in retro emulation, there are specialized distributions such as RetroPie (my RetroPie notes). However, there are plenty of gaming opportunities on the standard Pi operating system, Rasbian. Rasbian is a free and open-source OS, and most of the applications available are as well, which means no cost for you, so give these games a try!

Built in games

These can be found in the Menu, under the Games category:

Minecraft Pi: RPI ships with a special edition of the popular game Minecraft with a programming console for learning Python. Check out Getting Started with Minecraft Pi.
Python Games. A set of a dozen simple games writen in the Python programming language. This means you can play the games–and easily learn to edit them yourself. The source code for each is in /home/pi/python_games. Check out Al Sweigart’s Making Games with Python & Pygame to learn Python while making games!

Install packages

Linux distributions have package managers designed to automatically install and update in software in a safe and secure way. Rasbian is based on Debian linux and uses the apt utility on the command line. To use it, open Terminal, and type sudo apt install package-name (replacing package-name with the actual application name). You can search for package names using sudo apt-cache search. If you want more information, check Linux’s built in manual by typing man apt.

Debian’s repositories have a lot of fun games (here’s a decent list for Pi, or a database of all linux games). Keep in mind not all will run well on Pi given the low spec hardware, but it is all free, so why not give them a try? (p.s. remove packages using sudo apt remove package-name) Here are a few options:

Classic KDE games: kblocks kbreakout kapman kmines kgoldrunner kdiamond kspaceduel kpat
Arcade-like classics: frozen-bubble, xgalaga, snake4, battleball, tint, freesweep, pingus, chromium-bsu
Classic point-n-click adventures: flight-of-the-amazon-queen, beneath-a-steel-sky, drascula, lure-of-the-temptress
Tux classics: supertux (Mario inspired platformer), supertuxkart (Mario kart like races, might not work on current Rasbian), extremetuxracer (downhill penguin race)
Rogue-like: games-rogue (group of rogue-like games, including angband, crawl, moria, and slashem), crawl-tiles (classic rogue-like with graphics)
Others: freeciv (Civilization builder), openttd (Transport Tycoon), lincity (sim city), endless-sky (space exploration), freeorion (space conquest)
Create levels: there are many games inspired by commercial classics that fans create open engines and free content for, such as freedink (zelda-like),flare (action rpg), or freedoom (free DOOM game base).
Cat that follows your mouse: oneko (is this a game? well it’s amusing anyway…)

Web emulator

The Internet Archive has thousands of old games preserved in their digital library. They make them available in a browser based emulator, so you can play them anywhere, but Pi is just right for many of these retro games! Yes, you can play Oregon Trail right now, or check out these collections:

Commandline fun

The Terminal is a powerful command line interface in Linux. So why not use it for some fun?

Classic BSD Games: install bsdgames to get access to dozens of classics. Type the name of the game in the terminal to start playing, for example, tetris-bsd. Check the list for the names.
Text adventure: try the original colossal-cave-adventure
Rogue-like: enter the retro world of text-based RPGs with nethack, slashem, omega-rpg, crawl, angband, or moria
Wise cow: install cowsay and fortune to access the wisdom of the command line. Try cowsay "Hello" or fortune, then advance to fortune | cowsay or fortune | cowsay -f dragon.

OpenGL

On rpi2 with the newest version of Raspbian Stretch, you can enable a new hardware OpenGL driver. Run sudo raspi-config, look in Advanced > GL Driver. Select the new driver, then select finish and reboot the pi.

Try it out with a few OpenGL games, such as neverball and oolite.

Dates in Excel

Tue, 05 Sep 2017 00:00:00 +0000

Excel will mess up your dates. Assuming you want to use your data outside of Excel, it is best to avoid allowing it to convert them to its built in format, so import as “text” and set the column type to “text” to avoid issues.

Excel uses the 1900 date system (sometimes called numeric dates, serial dates, or 1900 system). It simply counts the number of days since 1900. Thus, 1900-01-01 = 1, 1900-01-02 = 2, and so on. This allows dates to be more easily used in calculations. However, MS has several historical bugs built into Excel which they have never fixed for compatibility reasons:

Some versions of Excel on Mac used 1904 date system
Excel assumes 1900 is a leap year, which is false.

So sometimes you will see plain text data, such as a CSV, exported from Excel that has dates that look like 42291 and you need to convert that serial date back into a real one. Unfortunately, unless you know what version of Excel it was exported from, you can not be sure if the serial date number is off by one day or potentially more (1904 system). But you can do you best to convert it back using OpenRefine GREL like:

"1900-01-01".toDate().inc(value.toNumber()-2,'day').toString('yyyy-MM-dd')

Start at Excel’s starting point, “1900-01-01” and convert to a date. Then use inc function to add the number of days in Excel’s serial date, but add -1 day because our date “1900-01-01” is actually 1 (i.e. we are not starting at zero), and then -1 again for the 1900 leap year bug. I add the cell value to the express with value.toNumber() if the column is currently a text column, but if it was already a number column, you could just use value. Finally, convert the date format into a preferred text format using toString().

Ugly.

Epic Music!

Sat, 02 Sep 2017 00:00:00 +0000

ABC notation is a shorthand standard for representing music, commonly used to record and share traditional folk tunes. Each tune has a header and some notes.

Common header elements:

X: index number, required in some software [e.g. X:1]
T: title [e.g T: Happy Birthday]
M: meter, required [e.g. M:C, M:4/4, M:3/4]
L: default note length [e.g. L:1/2, L:1/4, L:1/8]
Q: tempo [e.g. Q:120, Q:1/8=120, Q:1/4=60]
K: key signature, required [e.g. K:G, K:Gm]
V: voice, can set the clef [e.g. V:1 name=Violin, V:2 name=Cello clef=bass]

Common notation elements:

bottom octave CDEFGAB, top octave cdefgab [e.g. C = middle C]
z rest
, down an octave, ' up an octave [e.g. C, or c']
^ sharps, _ flats [e.g. ^c or _c]
make note longer by multiplying number, or shorter by a / dividing number [e.g. a2 or a/2]
notes separated by spaces will not have beams, those together will [e.g. a b c versus abc]
| barline, |] double bar, |: repeats :|

Epic Music reads Samuel Butler's translations of the Iliad and Odyssey as if it was ABC notation, a simple plain text music format. The editor is created using abcjs, which renders the ABC in musical notation and generates MIDI sound playback (full functionality only on Chromium / Chrome browser). Click the Edit button to show the ABC notation, and try modifying the Epic Music!

Brand New Reel!

Thu, 24 Aug 2017 00:00:00 +0000

ABC notation is a shorthand standard for representing music, commonly used to record and share traditional folk tunes. Each tune has a header and some notes.

Common header elements:

X: index number, required in some software [e.g. X:1]
T: title [e.g T: Happy Birthday]
M: meter, required [e.g. M:C, M:4/4, M:3/4]
L: default note length [e.g. L:1/2, L:1/4, L:1/8]
Q: tempo [e.g. Q:120, Q:1/8=120, Q:1/4=60]
K: key signature, required [e.g. K:G, K:Gm]
V: voice, can set the clef [e.g. V:1 name=Violin, V:2 name=Cello clef=bass]

Common notation elements:

bottom octave CDEFGAB, top octave cdefgab [e.g. C = middle C]
z rest
, down an octave, ' up an octave [e.g. C, or c']
^ sharps, _ flats [e.g. ^c or _c]
make note longer by multiplying number, or shorter by a / dividing number [e.g. a2 or a/2]
notes separated by spaces will not have beams, those together will [e.g. a b c versus abc]
| barline, |] double bar, |: repeats :|

Brand New Reel remixes the titles and tunes of about 460 traditional reels. The music data is derived from the ABC version of the Nottingham Music Database. Because ABC notation is a simple plain text format, it is ideal for sharing music on the web--and for manipulating with code. The editor is created using abcjs, which renders the ABC in musical notation and generates MIDI sound playback (full functionality only on Chromium / Chrome browser). Click the Edit button to show the ABC notation, and try modifying the Brand New Reel!

Open Music Theory Intro

Tue, 15 Aug 2017 00:00:00 +0000

Mini intro to working with OER “text book”

Welcome to Inquiry-Based Music Theory, OER Text Book!

What is it?

part of the Think Open fellowship
an OER
a fork (i.e. a new adaptation) of Kris Shaffer, Bryn Hughes, and Brian Moseley, Open Music Theory, Hybrid Pedagogy 2014+ (github repository).
a git repository hosted on GitHub. [more on Git and GitHub: Get Git, Git for collaboration]
a website build using Jekyll static site generator, hosted by GitHub Pages. [ more on gh-pages: go-go gh-pages ]
content written in Markdown, a simple mark up language written in plain text. [more on Markdown: markdown in a minute]

Activities:

GitHub account set up
practice repository
practice markdown editing
“commit”
fork

Workflow?

Epic Tweets

Wed, 02 Aug 2017 00:00:00 +0000

Homer @realEpicAuthor

Epic Tweets remixes Samuel Butler's translations of the Iliad and Odyssey into 140 character outbursts.

Epic Episode

Wed, 05 Jul 2017 00:00:00 +0000

Epic Episode remixes Samuel Butler's translations of the Iliad and Odyssey.

Get started with RetroPie

Mon, 26 Jun 2017 00:00:00 +0000

RetroPie is a fun and handy linux distribution that turns your raspberry pi into a console game emulator. It is intended to get emulators and controllers setup with minimum configuration and hassle. Perfect for your living room, the graphical interface is is designed to be navigated with game pad controllers, so you don’t need a boring old keyboard and mouse to use it!

Check the official installation guide for full details, but here is my quick reference summary:

Hardware

Suggested:

a Raspberry Pi 3 (from MCM, official distributor in US, also available many other places)
an micro SD card > 8GB (the RetroPie image is 2GB+, but you want plenty of room to add games)
some game pads, USB or bluetooth

Burn SD

Download the RetroPie image for your RPI type (it will be an img.gz file, you do NOT need to extract it).
Download and install Etcher if you don’t have it.
Insert your SD card into your computer and start up Etcher.
Select the RetroPie image (e.g. retropie-4.4-rpi2_rpi3.img.gz), your SD card, and click “Flash!”

For more about burning SD images, see Burn SD or USB note.

Start RetroPie

Put your SD into the RPI, plug the HDMI into a TV, and connect USB controllers.
Plug in the RPI and watch it boot. It will need go through some initial set up, such as expanding the file system–so this is a slow process, but only happens this once.
Follow on screen instructions to configure game pads. Tip: if configuration asks for more buttons than your device has, hold down one of the buttons to skip.

After configuring the pads, you will see a menu, but no games will show up, because you haven’t added any yet…

(personally I don’t bother connecting to wifi, but you could)

Add ROMS

There are a few methods to get ROMs on your RetroPie, but the most foolproof way seems via their USB method:

Get a USB drive with enough space to hold all your ROMs, preferably one that has a LED indicator light.
Format the USB stick.
Create a single folder named retropie on the USB stick.
Plug the USB stick into your RetroPie, watch the busy indicator light and wait for it to stop flashing.
Unplug the USB stick from RetroPie (no need to “eject”, just pull it out), and plug it back into your computer.
Copy your ROMs into the correct folders in the directory structure set up by the RetroPie (i.e. inside retropie/roms… keep in mind the folders are named after the international model names, so don’t necessarily match USA console names.)
Plug the USB stick back into the RetroPie, watch the indicator light and wait for it to stop flashing–this may take awhile depending on how many ROMs you are adding!
Restart the RetroPie via the menu system (access by clicking “Select + Start” on a game pad. Do not shut down your RPI by simply unplugging if possible, this has the potential to permanently corrupt your SD. When you give the shutdown command, wait for the screen to go blank and the green light on the RPI to stop blinking.)

However, my favorite fast method is to just pull out the SD and plug it into a linux computer. Then head into the /retropie/home/pi/RetroPie/roms directory, and copy roms into the correct folders. Unmount from your computer, and plug back into Pi.

Play!

You should be good to go!

General menu controls:

A = select
B = back
Start = menu
Start + Select (in game) = exit back to main menu
Select + B (in game) = reset

To shut down: “Start” (for main menu) > “Quit” > “Shutdown System”

Sonnet Thing

Thu, 22 Jun 2017 00:00:00 +0000

Sonnet Thing uses JS to take the text from Shakespeare's Sonnets, chop each line in half, shuffle it all, and spit out a new poem. Each "sonnet" has 14 lines, reconstituted from halves. White space and blank lines are included for the sake of variety.

Wget and Web Archiving

Wed, 07 Jun 2017 00:00:00 +0000

Mini intro to practical Wget for archivists

What’s in a URL?

https://example.com/about?key=value#anchor

protocol + domain name (optional port :80) + path + query with parameters + fragment/anchor

A subdomain can be added in front of the main domain name(s). For example, in lib.uidaho.edu > lib is a subdomain of uidaho, which is a subdomain of the top-level domain edu.

Dynamic vs Static web

A static website is a collection of HTML, CSS, JS, images, and other files that are delivered exactly as they are on the server to users. A URL in a static site generally represents a request for an HTML document in a specific file location.

Dynamic web uses a server-side scripting language to create pages on the fly when a user makes a request. Thus a URL represents a query, rather than an existing document on the server. Content, templates, and metadata are usually stored in a database. For example, WordPress uses the scripting language PHP and database MySQL. This enables more complex interactivity such as comments, customized views, user management, and a web-based admin interface.

Web crawls harvest the set of pages generated by following the links within a domain. This is a static snap shot at a specific point in time, meaning the dynamic functionality of a website will NOT be captured. Depending on the site design this could lead to loss of information. For example, some information is not fetched until a button click, links are written to the page using JS, images are changed for different browsers, or data is retrieved via a web form (POST request).

In a web archive features such as search bar, streaming media, widget embeds, and complex JS won’t work (or will introduce context anomalies). (example)

Webrecorder is a tool used to capture features that require user interaction, but this requires actually surfing everything you want to harvest. (example use case, Transparent Idaho)

Dynamic requests issues

UIdaho uses the CMS platform sitecore. This is a database driven dynamic site written using ASP.NET (?, check their SiteCore demo site).

Look at https://www.uidaho.edu/, notice that the hyperlinks on page look like https://www.uidaho.edu/academics.aspx. Rather than a static HTML document, the links are a dynamic request to “active server page extended” script, which creates the page https://www.uidaho.edu/academics (which is not academics.html or academics/index.html).

This causes problems for a web archive, since we harvest the resulting static HTML, not the aspx script. The file “www.uidaho.edu/academics.aspx” will not be in the web archive, but the document is captured as “www.uidaho.edu/academics”. (example)

Wget Prep

Wget is a handy Free command line tool to robustly retrieve documents from the web. It is a standard utility on Linux. On Windows, I suggest setting up a Bash terminal with Wget, for example Cygwin as outlined in Using Cygwin (note: I previously suggested Cmder as a handy portable option. However, we have discovered some bugs when creating WARC files with Wget on Cmder.).

See: Intro to the Command Line

Basic Wget

Open a terminal and navigate to a test directory.

Wget commands typically take arguments and a URL. Arguments are set using a -- flag. There is a long and short of version of most (e.g. --help and -h):

wget --help

To retrieve a single web page or file, just add the URL:

wget https://evanwill.github.io/_drafts/notes/commandline.html

To get a list of files, create a plain text list of urls you want to download, one per line. Use the --input-file= option to pass that list to wget.

wget --input-file=download-file-list.txt

Adding the --recursive argument allows Wget to act as a web crawler, following links a page until everything in a domain has been downloaded. All assets will be downloaded in a directory structure mirroring the site organization. A crawl can be limited to a specific file type using the --accept option. For example, download all PDFs:

wget --recursive --accept=pdf http://site-with-pdfs.com/

When using --recursive, add --no-parent, --level=NUMBER, or --domains=LIST to limit your crawl:

wget -r -np -Apdf http://site-with-pdfs.com/services/workshops/resources/

Correctly scoping your crawl is important, spend time exploring the hierarchy of the site to ensure you will capture what you want, but not download the entire internet…

To get an entire web site, use the --mirror and --page-requisites arguments. This will recursively crawl the domain and collect everything needed to reproduce the site. Adding --convert-links will rewrite the internal links to work offline if desired. It is important to add --wait=SECONDS and --random-wait to avoid bothering servers (you are unlikely to overload them, but they are likely to block you).

wget -mpk --wait=5 --random-wait https://example.com

Archival Wget with WARC

wget --help | grep warc

WARC is a web archive format that stores page content, response headers, and metadata for a group of web pages. One WARC can contain all the pages gathered during a web harvest. In addition to HTML documents, it can contain binary content such as images.

Wget can create a WARC for any crawl simply by adding the flag --warc-file="filename" to the command. Wget will harvest the site assets as normal, but additionally create a WARC compressed as a gzip file (.gz). For larger sites it’s a good idea to add --warc-max-size=1G to limit the max size of each WARC so they don’t get too big.

If the server refuses to give content to wget’s default user agent (sometimes identified as a robot), you can send a different one, like --user-agent=Mozilla. Occasionally it may be necessary to ignore “robots.txt” for archival purposes. Add --execute robots=off to the command.

Test example:

wget --mirror --page-requisites --wait=2 --random-wait --no-parent --trust-server-names --warc-file="test-archive" http://www.example.com/path/sometopic/

More complete example:

wget -mpkE --span-hosts --domains=example.com,www.example.com,sub.example.com --warc-file="test-archive" --warc-max-size=1G --warc-cdx --user-agent=Mozilla -e robots=off --wait=2 --random-wait http://www.example.com

More examples in use:

wget -mpkE -np --trust-server-names --warc-max-size=1G --warc-file="test-archive" --warc-cdx --wait=1 --random-wait https://www.example.com/path/news/newsletter

Limit using --include-directories= (instead of --no-parent), be sure to include all directories for page requisites:

wget -mpkE --trust-server-names -I /~,/css,/fonts,/Images,/Scripts,/path/news/newsletters --warc-max-size=1G --warc-file="test-archive" --warc-cdx --wait=0.5 https://www.example.com/path/news/newsletters/

Playback

Options:

Webrecorder Player (actively developed desktop app created by Webrecorder / Rhizome. Currently seems extremely slow and buggy on some computers when using warc not created by Webrecorder.)
Webarchive Player (not actively developed, but still works. simple desktop app)
pywb (“Python WayBack” Python package. Lots of features using wb-manager commandline utility. Can serve and manages multiple warc files in a collection.)

Workflow questions

Policy
Harvest
Store
Catalogue
Access

Reference:

ArchiveTeam, “Wget with WARC output”.
IIPC Awesome Web Archiving.
IIPC overview.
Corey Davis, “Archiving the Web: A Case Study from the University of Victoria”, code4lib 26 (2014), http://journal.code4lib.org/articles/10015.
Adrian Brown, Archiving websites : a practical guide for information management professionals (2006). http://search.lib.uidaho.edu/UID:everything:CP71127753580001451
Maureen Pennock, “Web-Archiving”, DPC Technology Watch Report 13 (2013), http://www.dpconline.org/docman/technology-watch-reports/865-dpctw13-01-pdf/file
Jinfang Niu, “An Overview of Web Archiving”, DLib 18, 3/4 (2012), doi:10.1045/march2012-niu1.
“Archivability” guide, Stanford Libraries
ArchiveReady (test sites for archivability)
Oldweb.today (surf web archives in emulated historic web browsers)

Collection Development Policy examples:

Columbia University Libraries, “Web Resources Collection Program”
Stanford Libraries, “Collection development”
MSU Archives, “Web Site Collection Plan” (Web Archives @ MSU)
Michael Shallcross, “On the Development of the University of Michigan Web Archives: Archival Principles and Strategies” SAA Campus Case Studies 13 (2011) http://files.archivists.org/pubs/CampusCaseStudies/Case13Final.pdf.

Tools:

Web Curator Tool
NetarchiveSuite (vagrant, package to manage harvesting developed by The Royal Danish Library)
WAIL (GUI interface to work with Heritrix and OpenWayback, buggy in my experience, repo)
WARCreate (Chrome plugin for one off WARC creation)
Wpull and grab-site (archival focused Wget alternative in development)

Videos:

Handy OpenRefine Functions to Remember

Thu, 01 Jun 2017 00:00:00 +0000

This page lists some handy functions to use for data wrangling tasks.

Combining columns

Combining columns can be tricky because merging a blank cell cell with another value results in an error. To avoid issues, first facet by blank and combine only non-empty cells with a transform like: value + " " + cells["col_2"].value

Cross function

Use cross to retrieve columns from other OpenRefine projects based on a common column.

Open the two projects you want to join
Identify the common column to be used as a key. Ensure that it is a unique identifier, or cross won’t work how you expect. You can create a new unique column on both projects by adding an index number an existing common column (on common column > Add column based on this column… > value + "-" + row.index ).
On the project where you want to import values, go to the key column you identified, and Add column based on this column…
In the expression window, put: cell.cross("name_of_other_project", "name_of_key_column").cells["name_of_column_you_want_to_import"].value[0]

You should have a new column that has the correct values from the other project.

String + Array functions

A powerful way to interact with multi-valued text fields (values with a separator in them, e.g. dogs; muffins; cats; idaho) or large strings (such as the text of poems or web scrape) is to turn them into arrays, then use array functions to manipulate. You might be surprized by how useful it is to break text values into arrays!

Create an array from any string by using the split(value, expression) function. The expression is the character or pattern you want to split the string up on, often a new line or a deliminator in a list.

For example, split on semi-colon value.split(";") (a classic multi-valued cell list), split on spaces value.split(" ") (basic word array), or split on a new line value.split(/\n/) (lines of a text).

Once the cell is an array, it can be rearranged and sliced in many ways with array functions. Finally, reconstitute the string by using join() on the array (usually using the same deliminator that you used to split!).

For example, if we had a column with lists of tags like “dogs;cats;muffins” as cell values, we could put each cell in alphabetic order using:

value.split(";").sort().join(";")

Remove the first item in the list:

value.split(";").slice(1).join(";")

Remove the last item in the list:

value.split(";").slice(0,-1).join(";")

Remove duplicate values in the list:

value.split(";").uniques().join(";")

Or trim the white space for each value:

forEach(value.split(";"),e,e.trim()).join(";")

If you had lines of a poem or text as a cell value you could do the same types of operations. For example, remove the first line of a poem:

value.split(/\n/).slice(1).join("\n")

Remove the last two lines:

value.split(/\n/).slice(0,-2).join("\n")

Or trim the white space around each line:

forEach(value.split(/\n/),e,e.trim()).join("\n")

De-dupe Rows

Deduplicate rows using the values in a key column:

On the key column to deduplicate, click “Sort”, and choose sort method.
Next to the show rows selection above the table, click on the “Sort” menu (this menu only shows up once you add a Sort). Select “Reorder row permanently” (if you do not do this step, sort is just visual and did not transform the data).
On the key column, select “Edit cells” > “Blank down”.
On the key column, facet on blank, select true (the blank values), and remove all matching rows.

Compare two columns

Create new “equal” column using expression: if(cells["column1"].value == cells["column2"].value, "True", "False")

Add leading zeros

If the column has numbers that should have leading zeros, add the number of zeros it should have in total digits, sliced by value length. For example, if you had “12345”, “123456”, “1234567”, and wanted them all to be 8 digits with leading zeros, transform using:

"00000000"[0,8-length(value)] + value

You can also create a new row identifier with leading zeros using the row.index variable. For example,

"row_id_" + "0000"[0,4-length(row.index +1)] + (row.index +1)

Remove leading or trailing character

In regex ^ is start of string and $ means end of string, which can be used in a replace statement.

Remove “T” from front of string:

value.replace(/^T/,"")

Remove trailing period, “.” at end of string:

value.replace(/\.$/,"")

(note the “.” needs to be escaped with \ since it has a meaning in regex)

Sometimes you have a column with many repeating values, that you might explore using a text facet. In the text facet pane you can sort by facet count, but you would have to manually select each if you wanted a subset based on the facet count. To select a group of rows based on the facet count of a value in a column:

First, if you just need all the values with > 1 count, you can use the built in Facet > Customized facets > Duplicates facet. This returns “true” for rows with > 1 count, false if the value is unique.

Second, if you need a subset based on the count, create a new column using the facetCount function. On the column you want a count for, Edit column > Add column based on this column, and use:

value.facetCount("value","name_of_the_column")

The result will be a number (same as the “count” given in facet pane), which you can then filter with a numeric facet. Note in this context facetCount seems a bit non-intuitive since you have provide “value” and the name of the column again–facetCount is set up with flexibility to do some more complicated operations by adding an expression to the value or matching values in a different column (but weirdly requires quotes around the expression which isn’t typical of other functions).

The “name_of_column” could be a different column than the one the cell is in. If you are using the current column, you can use the builtin variable columnName instead, like value.facetCount("value",columnName).

If you want to add transformations to the value first, apply the same to the value feeding in as the first argument.

value.fingerprint().facetCount("value.fingerprint()",columnName)

Parse JSON

It is common to get JSON data when fetching from APIs using Refine. It’s easy to grab specific dictionary values out of JSON cells using the built in JSON parse function. From the column with JSON, create a new column and transform with value.parseJson().get('key'), where ‘key’ is the dictionary key you want to extract.

For example, if the cell contained { "type" : "dog", "color" : "brown", "size" : "large" }, and your transform wasvalue.parseJson().get('color'), you would get the value “brown” in your new column. (note: if your key does not have spaces, you can use the shorter version like value.parseJson().color)

To get multiple values from the same key, combine with forEach(). For example, to extract all the keywords from a cell with the JSON {'language': 'en', 'keywords': [{'text': 'dogs', 'relevance': 0.979292}, {'text': 'muffins', 'relevance': 0.977987}, {'text': 'cats', 'relevance': 0.969001}, {'text': 'idaho', 'relevance': 0.967973}] }, transform with forEach(value.parseJson().keywords,v,v.text).join("; "), resulting in the new cell value of dogs; muffins; cats; idaho.

Parsing CONTENTdm TSV export

CONTENTdm and some other platforms export metadata in TSV format which often end up with parsing errors on import. When starting a project:

make sure you select the correct encoding (for CONTENTdm = “UTF-8”)
uncheck the option Use character " to enclose cells containing column separators
parsing issues are often not immediately apparent, so carefully check the number of records you expect and view the last rows of your data

Local server to input data from files

A goofy approach to get a bunch of text data into a spreadsheet from individual files is to serve the directory of files up on a local server then grab them using Refine’s fetch. This avoids many limitations of working with conventional spreadsheets, and lets you parse the files into data using Refine methods such as parseHtml, split columns, or split multivalued cells.

For example, imagine I have a folder of hundreds of HTML files that I want to parse into data:

create a list of the files on commandline with ls > list.txt
create Refine project using list.txt so each row will equal one of the files
start a local server in the folder of files and note where it is served (e.g. localhost:8080)
Add column based on the filenames with the local url, e.g. "http://localhost:8080/" + value
Add column by fetching urls

Now you have a spreadsheet with a giant amount of text data!

Common HTML parsing

Combining “Create new column by fetching from URL” and the parseHtml() GREL function is a powerful and flexible method to harvest data from the web or scrape sites. Always remember to use .toString() or .join("|") at the end of your parsing statements or you will end up with empty cells even through your html parsing is correct!

I often use these GREL statements to extract stuff out of HTML:

get all image src out: forEach(value.parseHtml().select('img'),i,i.htmlAttr('src')).join("; ")
get all links out: forEach(value.parseHtml().select('a'),i,i.htmlAttr('href')).join("; ")

Info on using more memory

https://docs.openrefine.org/manual/installing/#increasing-memory-allocation

Linux issue with Java version not found (V<3.0)

If you have an up-to-date version of Java installed on linux (openjdk-11-jre), starting Refine V<3.0 with ./refine will throw an error saying you don’t have Java installed or need to set $JAVA_HOME. The message is false (no matter how much you mess with $JAVA_HOME it won’t work!), there is a bug in ./refine where it checks for older versions of Java as “up-to-date” and was not future proofed.

To fix the bug, you can try a newer version of Refine (it is fixed in 3.0+), or just grab the updated ./refine file and manually replace your faulty one:

in your openrefine directory (e.g. openrefine-2.8/), delete refine.
Go to this updated version, click “Raw”, save as “refine” in the openrefine directory.
Done!

Transfer iPhone to Linux

Thu, 01 Jun 2017 00:00:00 +0000

There was a brief time when you could simply plug your iPhone in to your Linux computer via USB and access the photos. For all logical people this seems like sensible functionality. So of course, Apple killed it with the latest iOS 10 updates (cause Apple hates users, linux, and openness as far as I can tell based on my experience).

But maybe you are like me, you have an iPhone that some one gave you and you need to get the photos off. Don’t mess around with searching years and years of forum posts for solutions, because iOS changes constantly and intentionally prevents the functionality you want and expect. If you have iOS <=9, Ubuntu will work with it automatically. If you have iOS >=10, as of around March 2017 none of the old solutions to directly connect via USB will work. (*Note: as of 2020, it sort of works again, but see my notes about HEIC transfer issues)

The utility package libimobiledevice that allowed connection to iOS ships with Ubuntu now, but doesn’t seem to work alone with iOS 10+. Some have suggested downloading the most recent source code and manually building libimobiledevice works. However, with most i-devices, adding the simply ifuse package seems to bring back normal plugin functionality, but you have to manually mount the iphone’s drive.

Install it via sudo apt install ifuse. Then you can use ifuse to mount the iPhone file system:

make sure it is connected: idevicepair validate
create a mount point: mkdir ~/phone
mount the phone’s file system: ifuse ~/phone
now you can navigate to the directory and copy files off the phone (images are in “DCIM”)
unmount the iphone: fusermount -u ~/phone

Now if that didn’t work, oddly enough, the easiest solution seems to be FTP:

On the iphone, install FileExplorer by Skyjos.
You can’t directly share your photo library, so using the FileExplorer app, create a new “local” folder and copy your photos to it.
On the Local tab, click the wifi icon to start a FTP server.
Using the IP address and port given by the app, connect from your computer. The url is basically ftp://[ip-address]:[port]. On Ubuntu the easiest method is to use the built in “connect to a server” feature of the Nautilus file explorer. Open the file browser, click on “Other Locations”, look at the bottom of the window for “Connect to Server”. Type in the url and click connect. Now you can copy & paste where ever you would like!

Git notes

Thu, 01 Jun 2017 00:00:00 +0000

Quick fix the last commit

If you just made a commit, and haven’t pushed it yet, but suddenly realize you forgot to add something, need to change the message, or tweak a file, use git commit --amend.

make the changes you need
git add any changes (nothing if you are just changing the commit message)
git commit --amend -m "new commit message that replaces the old one"

This commit will just be combined with the last one into a single new commit as if the first one never happened.

Undo one file

Checkout the version of the file you want from the history, then commit that file. For example, go back a version from one commit back:

git checkout HEAD~1 example.txt
git add example.txt
git commit -m "undo example changes"

Undo a whole commit

git revert <somehash> will undo a commit you list by creating a new commit reversing those changes. It doesn’t go back to that point in the history or undo other commits along the way, it just does exactly the opposite of that one commit. Get the hash for the commit from the GitHub commits page or git log --oneline and add it to the end of git revert.

Undo a merge

You can undo a merge commit, which will undo the whole batch of changes created by the merge–the exact opposite of the merge.

However, when reverting merge commits, you need an extra option, because you need to tell it which side of the history you want to keep, i.e. the parent or the merging branch. Normally you would keep the parent with option -m 1, the command ends up looking like:

git revert -m 1 <somehash>

Your history will look wonky (and can potentially have odd effects if you try to merge the same branch again), but it is usually the best way to avoid problems collaborating on a remote.

Graph

You can actually reproduce those pretty visualizations of history you see on GitHub in the terminal. Try: git log --graph --abbrev-commit --date=relative

file naming for collaboration!

Be extra careful of file naming case sensitivity when working with people across Linux, Mac, and Windows.

Git is developed and designed for Linux. In Linux file systems, filenames are case sensitive. So files like “README.md”, “readme.md”, and “ReadMe.md” are three totally different files, and can be tracked and committed by Git. GitHub and other web services run on Linux so also allow case sensitive file naming.

This can lead to nasty Git errors if a Linux user commits files that would be the same if case insensitive. E.g. Linux user commits “README.md” and “readme.md” in the same directory, if Windows and Mac users pull the update they end up with fatal errors and a lot of confusion. It can’t be gracefully fixed on Windows or Mac, so it is best to clean it up on GitHub or Linux, then do a fresh clone (see xkcd git).

Git Merge Error Message

Recent Git installations (late 2023+) do not have a default behavior set for git pull, so the first time you encounter a conflict and merge, you will get a big error message asking which approach you want to use (merge or rebase). It looks like:

hint: You have divergent branches and need to specify how to reconcile them.
hint: You can do so by running one of the following commands sometime before
hint: your next pull:
hint: 
hint:   git config pull.rebase false  # merge
hint:   git config pull.rebase true   # rebase
hint:   git config pull.ff only       # fast-forward only

To avoid this error, you can set the old default behavior in your config, which is:

git config --global pull.rebase false.

Helpful

Rebase a Pull Request
About pull request merges
Rebase and merge
Amusing way to view Git Log, StarLogs
gitignore.io (ready made ignore templates for various project types)

Git in Classroom

GitHub

use Emoji in GitHub markdown, see Emoji cheat sheet

Kill Unresponsive Apps using Terminal

Tue, 30 May 2017 00:00:00 +0000

Lately on Ubuntu Firefox has been going unresponsive, then makes System Monitor go unresponsive when trying to kill it. Very frustrating, and I always wish I remembered how to kill processes on the command line.

Well, it’s not very hard, so hopefully I will really remember if I write it down here (since I can’t access my notes if Firefox is freezing everything)…

Basically, you need to know the process ID of the application, then send a kill command for that PID.

Find the PID, use one of these methods depending on what you know:
- top, simply type it into the terminal to get a visual display of the running processes and system resources, similar to System Monitor. The PID is the first column, and the offending unresponsive app is often at the top, using up 100% of one CPU core.
- ps x, will give a static list of all your processes (or ps aux for all users). You can sort through this using grep, like ps aux | grep firefox.
- pgrep, if you are in a hurry and know the offending app by name, try pgrep firefox to return a single PID.
Kill the process:
- first, try kill plus the PID, like kill 10705. This sends a polite message to the process to shutdown. If the app is unresponsive, it probably won’t work.
- then, force the kill by using the SIGKILL flag (-9 or -KILL), which directly tells the kernel to kill the process, rather than asking the app first. For example, kill -KILL 10705

Don’t forget, kill -KILL!

Create a GitHub Organization

Sat, 06 May 2017 00:00:00 +0000

A GitHub organization is a useful tool to collaborate with a group and ensure continuity in situations with members may move on from a project or lab. GitHub organizations are NOT a separate account or profile, but are a group of users, i.e. you can’t log in as an Org. One or more persons can be the Owner with full admin control over the org including managing members.

To start an organization, click the plus sign icon on the upper right of the GitHub menu bar and select “Create organization”. Give your org a good name, preferably with no spaces, since this will be used for your page and URLs. Select the Free account. Then add members!

Now, when creating a new repository, all members can use the drop down option to create the repository under their own account or under the organization. Repositories owned by the Org will be accessible and managed by the Owner accounts. They stay with the Org even if the original creator leaves the Org.

Accounts in your Org have one of two roles:

Owner = full admin access to everything
Member = can see all teams and members, and can create new repos

Your Org can create Teams. Set up teams to manage groups with access to specific repositories. Teams also have their own page that hosts a discussion forum, and can be @ mentioned to notify all members in other discussion comments.

People who need access to a repository, but aren’t part of the Org, are added as “Outside Collaborators”.

Education discount

Visit the GitHub Education page to learn about current discounts. In general, they grant free private repositories to non-profit, educational organizations for teaching purposes. Click the “Request a discount” button and fill out the request form will full information about your Org to get the discount applied.

Using Cmder

Sat, 06 May 2017 00:00:00 +0000

Out of date! I haven’t used this for a long time!

Cmder is a fully featured terminal emulator for Windows that can host a variety of shells, including Git for Windows Git Bash (which comes bundled with the full version). It is handy because it is full portable–simply unzip and start using it. Once you configure Cmder, you can even copy the directory to a USB and use/share your fully customized version. This is great for workshops.

“Install”

Download the full version and unzip to a permanent location (such as C:\).

In the cmder directory, double click Cmder.exe to get started. Windows might give a warning about programs downloaded from the internet that you have to Okay to run. ConEmu (the terminal emulator) might also suggest you download a newer version, click “Cancel” to ignore or the first download option to automatically install the update. Pin the icon to the taskbar to make starting it easier.

By default Cmder opens with the standard Windows cmd.exe shell, but you can open other shells in a new tab by clicking the drop down next to the plus icon in the lower right. Options include Windows CMD (bad), PowerShell (bad), mintty (better, same as Git Bash), or bash (best). You can (and should) change the default shell. Click the menu button in the lower right and select “settings”. Click on “Startup”, then check the “Specified named task” option and select “{bash::bash}” from the drop down, and “Save settings”. Now when Cmder starts, Bash will be the default terminal.

On Bash, your drive letters will be translated to UNIX-like paths. For example, Windows C:\ will be /c/. Your home directory will probably be /c/Users/username.

Similar to Git Bash, binaries for standard utilities can be placed the bin directory to add functionality. Anything in the bin will be on the PATH, i.e. you can use it directly on the command line. Sometimes the Windows binaries have funny filenames, so you should rename the .exe file to the standard name.

Add Nano editor

Unlike Git Bash, Cmder bash supports adding Nano. Get the Nano binary from Nano win32-support page. You just need the .exe file, which is named nano-git-0d9a7347243.exe (as of this writing). Download it, rename the file to nano.exe, and copy to the Cmder bin directory.

Now, Nano editor can be started from the Bash shell using a command such as nano test.txt.

Add Wget

Download the Windows binary from one of the options listed at Wget FAQ (for example, from eternallybored wget64.exe. Rename the file to wget.exe, and copy to the Cmder bin.

Note: I have noticed some bugs when using Wget on Cmder to create WARC files. For more complex use of Wget, consider Cygwin instead.

Add Ubuntu Bash shell

If you have Windows 10 and installed “Windows Subsystem for Linux”, you can add the “Bash on Ubuntu on Windows” shell to Cmder. Check solutions here and here.

Git for Collaboration Intro

Fri, 05 May 2017 00:00:00 +0000

Introduce using Git and GitHub for collaborating on code, writing, and websites.

Why use Git and GitHub?

Git is a free, distributed version control system originally developed for coordinating huge software development projects (i.e. the Linux Kernel led by Linux Foundation). However, Git is also great for personal uses such as organizing the code for your research project or the drafts and notes for writing an article. Pair Git with free hosting from Github and you have a powerful platform for managing your code and writing while collaborating and sharing with others.

If you are working with code or collaborating, version control WILL make your life better!

See:

Get Git, Why
Software Carpentry, Version Control with Git
also check out Data Carpentry for discipline specific data lessons

Basic Workflow

I want to introduce collaboration aspects of GitHub, but it’s easiest to explain by using it! Git seems pretty weird until you Do it…

Follow the first time Setup and Workflow from Get Git.

Skip the History section for now, but just keep in your heart that Git keeps track of Everything! Have peace of mind that you can’t loose your history once committed to a Git repo, unless you try really hard.

Collaborating via GitHub

Putting your project on GitHub adds powerful options for user management, project management, and sharing.

There is two basic workflows to collaborate on a GitHub repository:

Add collaborators (simple, typical of smaller projects)
- On GitHub, click the “Settings” tab of your repository.
- On the left menu, click “Collaborators”.
- Add collaborators via email or GitHub name.
- Collaborator will need to accept the invite.
- Clone the repo to your local machine.
- Now you all have equal control over the repo content: push, pull, merge, etc.
- Using feature branches may be helpful to organize your work–create a branch, do some work, push the new branch to GitHub, then create a Pull Request to discuss with your team.
Fork and Pull Request (more complex, centralized control, typical of bigger projects)
- Navigate to your partner’s repo on GitHub.
- Click “Fork” in upper right.
- Make changes in your personal fork of the repo.
- On your personal fork, click “New pull request” button.
- Check the changes, click “Create pull request” button.
- Create a message saying exactly what changes you made and why.
- The original repo will now have a PR that collaborators can view and comment on. Only the owner can accept the request and merge it.

Note: it is best to create small, targeted commits when collaborating. Each commit should do one specific thing making it easier for others to understand your work and navigate the history if necessary. PR’s should include only one change so they can be merged in as a module–don’t combine several unrelated commits since the owner may not want to merge all of them.

Project management features

GitHub adds many handy web-based features to manage your projects / Git repositories:

Issues. Create an issue to discuss and track ideas, bugs, projects, requests, etc. Can be assigned to people, tagged, and more. Also allows people outside of the project report problems with your code. Be sure to create checklists in the first comment box (- [ ] step)–they become click-able and show progress in the Issue view.
Projects. Create Trello board like lists to organize work.
Wiki. Simple wiki-style documentation that can be edited by your collaborators (note: written in Markdown not wikitext).

gh-pages

GitHub also offers free web hosting for your project, organization, or personal profile. Check workshop Go-go gh-pages!

Also, use GitHub Gist to instantly share simple notes, outlines, snippets, etc.

Resources

Markdown on Jekyll

Wed, 19 Apr 2017 00:00:00 +0000

Jekyll uses Markdown .md to simplify creating the base content for a site. The original spec is from Daring Fireball, intended to be simple and intuitive like writing an email. GitHub added extra features to the Markdown used for comments and writing throughout their site, called GitHub markdown flavor. Their Mastering Markdown Guide is a good quickie tutorial.

By default, Jekyll uses the Ruby based kramdown library to convert Markdown. Kramdown supports a few more extras beyond the standard syntax, such as attributes, footnotes, and typographic symbols. Check the kramdown syntax for details.

Tips

White space

Although Markdown is simple, it is important to remember that white space, blank lines, and tabs matter. If you are getting unexpected results when rendering, check your white space.

Add attributes to links

Kramdown allows you to add additional attributes to links. I often add a target, like this: [example](https://example.com){:target="_blank" rel="noopener"}.

Intermix with HTML

All HTML is valid Markdown, but by default Markdown converters ignore Markdown inside HTML tags. For example, <div> # Header One</div> won’t render as an <h1>. However, with Jekyll’s kramdown, you can add the attribute markdown="1" to an HTML element to tell the converter to render the Markdown inside. Thus, <div markdown="1"> # Header One</div> will render as <div><h1>Header One</h1></div>. This is very handy for adding special divs with ids and classes!

Nesting code blocks

I make lots of step-by-step tutorials. If you aren’t careful with code blocks nested in lists they will end up with extra spaces. The code block looks right, but when you copy and paste from the web page, they are all indented two spaces. The trick to fix this is that the code block has to line up with the first character of the list text, not with the list marker.

E.g., this works:

1. one
	- nest
	- nest
	  
	  ```
	  code
	  block
	  ```
	- more list
2. two

one
- nest
- nest
```
code
block
```
- more list
two

but this doesn’t:

1. one
	- nest
	- nest
	  
	```
	code
	block
	```
	- more list
2. two

one
- nest
- nest
```
 code
 block
```
- more list
two

OpenRefine POST request with Jython

Mon, 10 Apr 2017 00:00:00 +0000

It is easy to fetch links with OpenRefine using the Add column by fetching urls transformation. However, many APIs use HTTP POST to send information to the server for processing. Refine’s fetch can only HTTP GET.

To make a POST request from Refine you can use Jython in the expression window instead of GREL. Jython is included with Refine as a .jar file in the openrefine/webapp/jython/ directory. The current version is Jython 2.7 based on Python 2.7.

Jython in the Refine expression window requires a return statement to fill the new cells in the transformation. For example, the standard GREL value would be return value in Jython. The basic GREL variables use brackets instead of periods in Jython, for example GREL cells.column_2.value would be Jython cells['column_2']['value'].

To create a POST request use the standard libraries urllib2 and urllib. First, get your data into the correct format using urllib.urlencode(). Then, construct a request with urllib2.Request(url[, data][, headers]) and open it with urllib2.urlopen(req). Use read() to convert the object to usable data. If the data parameter is used in your request it will be a POST.

A basic GET request for a list of URLs in a column would be:

import urllib2
f = urllib2.urlopen(value)
return f.read()

To do a POST, construct the request first. For example, to get the sentiment of a line of text in each cell in a column:

import urllib2, urllib
url = 'http://text-processing.com/api/sentiment/'
data = urllib.urlencode({'text': value})
req = urllib2.Request(url,data)
post = urllib2.urlopen(req)
return post.read()

In most cases urlencode is not necessary, and in the tiny Refine expression window it is more pragmatic to write it in a compressed form:

import urllib2
url = 'http://text-processing.com/api/sentiment/'
return urllib2.urlopen(url,"text="+value).read()

If the API returns errors, try using GREL trim whitespace, GREL escape() / unescape(), or Jython encode("utf-8") And write a script that has error handling, like:

import urllib2, urllib
url = "http://text-processing.com/api/sentiment/"
data = urllib.urlencode({"text": value.encode("utf-8")})
req = urllib2.Request(url,data)
try:
    post = urllib2.urlopen(req)
except urllib2.URLError as e:
    if hasattr(e, "reason"):
        return "Failed: ", e.reason
    elif hasattr(e, "code"):
        return "Error code: ", e.code
else:
    response = post.read()
    return response

The official Refine Jython documentation is pretty sparse. Refine Jython includes the standard libraries, but others can be adding using a work around.

CONTENTdm api tips

Tue, 04 Apr 2017 00:00:00 +0000

Many libraries use CONTENTdm to host their digital collections. You may want to harvest metadata, text, and digital objects from these repositories. CDM is not the easiest to use, but there is a few handy ways to get stuff out via URLs.

We can access the CDM data using the CDM API. This API is helpful because its possible to create a set of queries using OpenRefine or a language such as Python to add the harvesting raw materials step directly to your data processing workflow.

CDM Terminology

The key to using the CDM API is to figure out the base URLs, the collection “Alias” (the short name represented in a collection’s url), and the item id “pointer” (the id number assigned the item by the CDM database). (Note: very old CDM instances also required a port, usually “:81”, that is no longer necessary, but still often appears in CDM documentation)

CDM uses some terminology for the components of their API:

CDM server number: each hosted CDM has a number which is used in two different URLs in the “contentdm.oclc.org” subdomain, which are used for different types of API calls.
- “CDM server URL” - this is where your CDM admin is hosted, named “server” + server number, e.g. https://server12345.contentdm.oclc.org/. In general collection level API calls have to go to this URL.
- “CDM website URL” - this is where your public CDM instance is exposed, named “cdm” + server number, e.g. https://cdm12345.contentdm.oclc.org/. Many organizations have a custom domain in front of this subdomain which should work for most API calls. In general item level API calls have to go to this URL.
“Alias” - the identifier name for an individual collection, used in URLs. If set by CDM they look like p17254coll3, or can be manually created looking like argonaut.
“pointer” / “CONTENTdm number” - the identifier number of an individual item or object assigned by the CDM database, e.g. 14. If harvesting metadata, this value is usually called “pointer” in XML. However, it is found in metadata export as “CONTENTdm number”.

If you look at the URL for at a CDM item page you can find the information. For example, in:

https://cdm17254.contentdm.oclc.org/digital/collection/p17254coll3/id/155/rec/1

The “CDM website URL” is https://cdm17254.contentdm.oclc.org, collection “Alias” is p17254coll3, and “pointer” is 155.

If you know the “CDM website URL” using the “contentdm.oclc.org” subdomain, then you can find the “server url” by replacing the leading “cdm” with “server”, in this example https://server17254.contentdm.oclc.org/. If you know a “server URL”, you can use the API for find alias and pointers. Unfortunately, it is difficult to figure out the server url if the organization uses a custom url.

If you have direct access the the CDM instance–skip the metadata harvesting–instead export collection metadata from the admin interface, then use the “CONTENTdm number” it provides to harvest objects out of the database (see CDM metadata tips info).

Note: it is also possible to harvest metadata via OAI PMH–however OAI PMH is even more tedious that CDM’s API and does not contain all the information necessary to harvest digital objects. In my experience, CDM’s implementation of OAI PMH is unreliable with larger collections.

Collection level API

These calls require the OCLC server url, e.g. https://server12345.contentdm.oclc.org/ as used in the examples below.

Find Alias and Pointers

Get an XML list of all collections using:

https://server12345.contentdm.oclc.org/dmwebservices/index.php?q=dmGetCollectionList/xml

(if you prefer JSON, replace /xml with /json)

From that XML document you can find the alias for a collection of interest in the <alias> element.

Next, you want a list of items so you can find the pointer. There is no obvious easy way. The best way is to choose a metadata field to query that every item has (generally you’ll be safe just using the field title to query). Following our example, to check which metadata fields exist for a collection with alias “example”, use:

https://server12345.contentdm.oclc.org/dmwebservices/index.php?q=dmGetCollectionFieldInfo/example/xml

This list will provide the list of fields including name and nick, you have to use the nickname in API calls using the field value.

After choosing a field, for example “title”, we can form a query to return items for the “example” collection. There are a bunch of options that must be included for the query to work, and if you don’t want to use the option you usually replace it with 0 in the url string.

https://server12345.contentdm.oclc.org/dmwebservices/index.php?q=dmQuery/example/0/title/title/1024/0/0/0/0/0/xml

Importantly, the API will only return a max of 1024 items. Note the 1024 in url above is the “maxrecs” value. To get more than 1024 items, you will have to do multiple API calls, changing the value directly after “maxrecs” (e.g. /1024/0/, then /1024/1024/, etc).

The first XML element <pager> tells you how many <total> items there are in the collection. Each item in the collection is in a <record> element with a <pointer>. Now that you have pointer for individual items, you can retrieve metadata or files for the items.

Get full text

Most document collections in CDM have a full text field filled by an OCR transcript. If you want to get the full text for all items in a small collection, its easy using the Query method mentioned above. The automatically populated full text field is usually called full in CDM. For example:

https://server12345.contentdm.oclc.org/dmwebservices/index.php?q=dmQuery/example/0/full/xml

However, this will only return 10 items. You can set the max number to return between 1 and 1024. So to set it to 1024, use this query:

https://server12345.contentdm.oclc.org/dmwebservices/index.php?q=dmQuery/example/0/full/nosort/1024/xml

Obviously, if your collection has more than a thousand items, this isn’t very helpful. However, CDM usually times out on large collections anyway, this method probably won’t work on larger collections with large full text fields. Instead, make a list of pointers, and use them to individually download metadata as shown below.

Get item metadata

To get an item’s full descriptive metadata, use the alias and pointer. For example, if the pointer is “1”:

https://server12345.contentdm.oclc.org/dmwebservices/index.php?q=dmGetItemInfo/example/1/xml

Item level API

For these calls you will need to know the CDM URL, alias of the collection, and pointer of the item. The url will work with either the OCLC CDM address, e.g. https://cdm12345.contentdm.oclc.org/, or a custom domain of an organization, e.g. https://digital.lib.uidaho.edu/.

In the examples below, use CDM url https://cdm12345.contentdm.oclc.org/, collection alias p12345coll1, and pointer 1.

GetThumbnail - will provide a thumb for any object in the repository, “/utils/getthumbnail/collection/alias/id/pointer” - https://cdm12345.contentdm.oclc.org/utils/getthumbnail/collection/p12345coll1/id/1
GetFile - provides a download link for PDF objects, “/utils/getfile/collection/alias/id/pointer/filename” - https://cdm12345.contentdm.oclc.org/utils/getfile/collection/p12345coll1/id/1/example.pdf
GetStream - provides download and playable link for video/audio objects, “/utils/getstream/collection/alias/id/pointer” - https://cdm12345.contentdm.oclc.org/utils/getstream/collection/p12345coll1/id/1

Use IIIF for Images

CDM now implements a IIIF API exposed at “/digital/iiif/”, only for collections that contain images. This is the best method for accessing images in collections.

Get max image size: https://cdm12345.contentdm.oclc.org/digital/iiif/p12345coll1/1/full/max/0/default.jpg
Get the full frame of an image at 50% size: https://cdm12345.contentdm.oclc.org/digital/iiif/p12345coll1/1/full/pct:50/0/default.jpg
Get image info: https://cdm12345.contentdm.oclc.org/digital/iiif/p12345coll1/1/info.json

Check IIIF Image API 2.1.1 docs for more possibilities. However, CDM’s implementation of IIIF has some limitations:

using the Size parameter !w,h does not work, instead returns max instead.
using the Size parameter pct: with less than 10% does not work, and will return an error message.
if either pixel size used in Size parameter w,h is larger than actual dimension of the full image, it will return an error message.

Using Cygwin

Thu, 09 Mar 2017 00:00:00 +0000

Out of date! I haven’t used this for a long time!

Cygwin is a project that adds Unix style terminal to Windows. It includes hundreds of Unix command line utilities built and packaged to function natively on Windows. It includes more functionality than Git Bash (included with Git for Windows), is open-source (unlike MobaXterm), and has a package system to install utilities (unlike Cmder). If you are currently using Git Bash on Windows, you may want to move to Cygwin if you would like Nano Editor, Wget, and GNU Make (plus old school terminal games!).

Install Cygwin

Download the current installation utility from the Cygwin site. I suggest the 64-bit version, setup-x86_64.exe. Once the file downloads, move it to a sensible permanent location as the utility is used to update and make changes to your install. For example, move it to C:\cygwin-stuff (by default Cygwin uses C:\cygwin64 as its root directory).

To get started quickly, simply double click setup-x86_64.exe to use the graphical installer. However, the awkward interface makes finding and selecting packages very difficult. I suggest starting on the command line, which greatly simplifies getting what you want / need.

Command line installer

Open Windows cmd command prompt in the directory containing setup-x86_64.exe. Enter the command:

setup-x86_64.exe -q -P nano,wget,git,chere,make,openssh,openssl,ninvaders,sl,fortune-mod,ctris

The -P option allows you to list the extra packages you would like installed. Get the package names from the full package list and separate each with a comma.

The command above will start the Cygwin graphical installer with the listed packages selected in addition to the core utilities. Since all the necessary options are not specified, the GUI will ask you a few more questions (check the full list of CLI options if you want streamline further).

Graphical installer

The installer will ask about these options:

Choose Download Source: “install from Internet”
Select Root Install Directory: the folder selected in this step with become your fake Unix style root directory /. The default is C:\cygwin64, which should work for most users. Do not change it to an existing system directory! All Cygwin files will be located in this root, so you will risk over writing existing files or causing incompatibilities.
Select Local Package Directory: this folder will be for package downloads. I usually change it to a new directory in the same location as setup-x86_64.exe.
Internet connection: use default unless you have a proxy.
Download Site: choose a mirror to download from. Any should work.
Select Packages: This is where you can add additional packages beyond the default install–if you started with the command line options above, everything you need is already selected. Otherwise, select “Full” from the “View” drop down menu, then search to find useful packages. Click on “Skip” to mark the utility for installation. The list is not very user friendly, but poke around until you mark off everything you need.

Update

To update or install new packages, run setup-x86_64.exe again. Your current packages and configurations will be saved, but everything will be updated to the latest versions.

Using Cygwin

The installer should put a shortcut on your desktop. Click it to start the terminal (this starts cygwin64/bin/mintty.exe, if you double click cygwin64/Cygwin.bat you will start in the awful Windows cmd.exe terminal instead).

File system

It is important to know that the file system is a bit wonky via the Cygwin shell. According to Cygwin, your / root directory is C:\cygwin64. Cygwin creates a home directory inside this root. This means that cd ~ then pwd will read /home/username, but it is NOT your Windows user directory. Instead it is C:\cygwin64\home\username, your user directory in Cygwin’s root.

To access your normal Windows file system use /cygdrive/. For example cd /cygdrive/c will bring you to C:\. Thus, your normal Windows user directory will typically be at cd /cygdrive/c/Users/username.

Right click

To make life a bit easier, use the chere package to add a Windows context menu item (i.e. right click).

To configure, right click on your Cygwin terminal shortcut (on desktop or in start menu) and choose “Run as administrator”. Then type the command: chere -i -t mintty. This adds the default terminal to your context menu as “Bash Prompt Here”. Simply right click any folder and select the option to open Cygwin in that directory.

The option can be uninstalled via Windows Control Panel like a program. To see the full options check chere --help.

CSS Animation

Wed, 08 Mar 2017 00:00:00 +0000

↓ →

Kind of amazing the stuff you can do with CSS with enough selectors…

Kids Apps on Linux

Sat, 04 Mar 2017 00:00:00 +0000

What if you give a little kid (3 - 6 years old) a Raspberry Pi?
Better add some free and open kids games!

Linux has had a lot of idealistic attempts to develop kid focused software. Unfortunately, most of the projects seem to fade away and are pretty dated today. Luckily, little kids really don’t care! They are tickled just to play with a computer. So, here is my list of apps that seem to successfully amuse little kids.

Essentials:

Potato Guy (KTuberling), sudo apt-get install ktuberling. Fun “stamp” game where you move stickers onto a scene such as a potato head or moonscape.
Tux Paint, sudo apt-get install tuxpaint. Easy to use paint program with Tux.
Bam Bam, sudo apt-get install bambam. Abstract infant game that is a lot of fun. Hitting any key or moving the mouse creates random visual and audio outputs filling the screen. The app locks to full screen, the only way to get out is to type quit.

Good:

GCompris, sudo apt-get install gcompris. A full suite of educational games, including activities that introduce using computer peripherals. Check the official site for more up-to-date packages.
Childs Play, sudo apt-get install childsplay. A suite of games for kids, some educational, some amusing. Official site.
PySio Games, sudo apt-get install pysiogame. Pygame based suite of games, see official site.
sudo apt-get install pysycache. Another Pygame based package for younger kids to learn the mouse.
Tux Type, sudo apt-get install tuxtype.
Tux of Math Command, sudo apt-get install tuxmath.
Gamine, sudo apt-get install gamine. Basic mouse-based “game” for very little kids.
Debian’s educational games, sudo apt-get install games-education. Many of the packages above can be installed at once with this metapackage. See full list.

Games

Classic games from KDE: kblocks kbreakout kapman
oneko cat follows your mouse
frozen-bubble
ri-li toy train sim

Fun with parent interaction:

Stellarium, sudo apt-get install stellarium. Night sky simulator with lots of options.
Celestia. 3D astronomy simulator. It’s not in many repositories right now due to a history of buggy-ness, but you can get it from the official site.
VirtualLab Microscope, http://virtual.itg.uiuc.edu/
xball, sudo apt-get install xball. Bouncing ball simulator.

Full distros

DouDou Linux is rather dated, but still works well as a totally ready to go, self-contained, live disk.
Sugarlabs unique kid focused distro originally developed for One Laptop per child project. Check On a Stick or on Ubuntu.
Ubermix is for a bit older kids, but has a clever install system to make restore seamless if users mess up the system.

Intro to the Command Line

Sat, 04 Mar 2017 00:00:00 +0000

Mini-workshop: Does command line interface (CLI), REPL, terminal, console, or shell sound scary? Don’t worry, we’ll learn about the basic computer interface that gives you super powers! If you want to work with Raspberry Pi, Linux, or many programming languages you need a basic familiarity with the CLI. This MILL-Mini will demystify the terminal so you can use it to get things done.

What is the Shell?

The command line is a text-based interface for efficiently getting stuff done. It’s often called the shell, terminal, console, cmd, or Bash, but ultimately the Shell is just a program on your computer like any other application. Its job is to act as a command shell, taking input from the user, ordering the computer’s operating system (OS) to execute the instructions, and returning the outputs.

Since the 1980’s the most common way to interact with a computer is via the graphical user interface (GUI), i.e. the windows, icons, and pointers of a desktop environment. Technically this is a shell. However, when people talk about the shell it usually refers to a command-line interface (CLI) made up of only text in a read-evaluate-print loop (REPL).

This means:

you type something on the input line and press return
the shell interprets the command
the OS executes it
the shell prints the output
repeat!

Note that the shell does not execute commands itself. It calls on hundreds of other applications in the OS to get things done. It’s the most basic way to interface with your computer short of re-wiring it!

Which Shell?

There are actually a lot of shells out there! Most OS come with one as a standard application. On Windows, CMD or PowerShell are normally available. These use a syntax and set of applications unique to Windows systems.

However, the most popular and commonly used is the Unix shell Bash (Bourne Again SHell) which is standard on Linux and Mac OS. Even if you are a Windows user, learning Bash will open up a powerful set of tools on your personal machine, in addition to familiarizing you with the standard remote interface used on almost all servers and super computers. If you want to control a Raspberry Pi or virtual machine in the cloud, you will need to use the Bash Shell.

On Linux or Mac, you already have Bash, just open a Terminal and you are ready to go. On Windows there are several options to add a UNIX-like style shell:

Windows Subsystem for Linux: Windows 10 has the option to install a Ubuntu Linux subsystem which enables a native Bash shell with all the standard functionality. This is very handy, but installing takes a few steps. (WSL notes)
Git Bash: installing Git for Windows adds the Git Bash shell. It comes with some of the standard UNIX applications (and you can add more). It is easy to install, integrates with Windows context menus (right click), and works well. (alternatively, use Git Bash with Cmder, which is fully portable, no install needed)
Cygwin: a project that adds Unix style terminal to Windows including hundreds of utilities built and packaged to function natively on Windows. It includes more functionality than Git Bash and has a package system to install utilities (more info).
MobaXterm: a non-opensource emulator similar to Cygwin with quite a bit of functionality and an optional paid version.

The GUI window that hosts the text based CLI is often called a terminal emulator. Most people use the default that comes with your OS, but if you want to get fancy, there are a surprising number of options out there! Check out Guake (linux, dropdown), Hyper (slick looking, html/css based), ConEmu (windows, tabbed), Tilix (linux, tiling terminal), Cool Retro Term (linux, retro emulator with flickering CRT screen look).

Navigate

Start up your terminal and type pwd.

This command asks the shell to “print working directory”. The shell should give you some out put, like /home/username. That’s where we are right now! The string /home/username is a file path representing our current location in the file system, just like navigating the folders in the GUI explorer. Typically, on Unix systems the / is the root directory. All other directories are organized below it.

Note: when using Git Bash on Windows, cd ~ should bring you to your Windows user directory, such as C:\Users\username. However, the Windows alternatives locate the / directories in a “fake” root directory, not the actual Windows file system root. For example, on Git Bash cd / will bring you to something like C:\Program Files\Git\mingw64\ not C:\. Cygwin and MobaXterm also place the “home” directory inside the fake root. For example, when using Cygwin, cd ~ the pwd will say /home/username, but will actually be located in the Windows C:\cygwin\home\username.

Take a look around:

ls (“listing”)
ls -a (-a is a flag, it adds an option to the command, in this case to show all files including hidden ones. There is often a short and long version, -a or --all)
ls --help (you can use --help with almost any command! Linux also has a built in manual, try man ls)
cd Desktop (“change directory”)
cd .. (back one level)
Save typing with tab complete and up arrow for command history!!
Copy & paste with your mouse or ctrl+shift+C / ctrl+shift+V (short cuts may need to be enabled in your terminal’s settings options)

Work with files

Let’s make a test directory to play in:

cd ~ (back to home directory)
cd Desktop
mkdir test (“make directory”)
cd test

Now lets create a file in it:

echo hello world (terminal repeats what you said)
echo hello world > test.txt (the > is redirect that prints the output to a file rather than the screen)
ls
cat test.txt (“concatenate” displays one or more files in the terminal)
echo more text >> test.txt (the >> redirects the output and appends it to the end of an file. A single > would over write the file)
cat test.txt

Play with it:

mkdir new
cp test.txt new/test.txt (“copy”)
cd new
mkdir new2
mv test.txt new2/test2.txt (“move” changes location and/or name, be careful!)
cd new2
rm -i test2.txt (“remove”, be careful there is no recycle bin!)
wildcards ? and *, try grep -i "hello" *.txt

Run Programs

Many programs run interactively inside the terminal (note: these don’t work well in Git Bash, but can be installed in Cygwin):

nano test3.txt (text editor in the terminal)
python (running another REPL system in the terminal)
curl http://www.lib.uidaho.edu/ (utility to get URLs in terminal)
top (process monitor)

Have Fun

A good way to get familiar with the commandline is with some fun and games! These demos install amusing programs (note: demos based on Ubuntu. Only available on Linux or Bash on Windows Subsystem. However, most can be found in Cygwin packages.)

Train:

sudo apt-get install sl (apt-get is Debian / Ubuntu’s utility to find and install packages)
sl

Wise cow:

sudo apt-get install cowsay
cowsay "Hello world!"
cowthink "Grass?"
sudo apt-get install fortune
fortune
fortune | cowsay (pipes | send output to another command)
cowsay -l
fortune | cowsay -f dragon

Banners:

sudo apt-get install figlet
figlet Command Line!
figlet -f slant Command Line! (more font -f options: slant, shadow, small, smslant, smshadow, big, digital, bubble, script, smscript, lean, block, mini, banner)

Classic games:

sudo apt-get bsdgames
tetris-bsd (see full list)

Matrix:

sudo apt install cmatrix
cmatrix (try -r, -b or check options with -h)

Watch Star Wars: telnet towel.blinkenlights.nl

Read some xkcd comics!

Solve a mystery…

Reference

SS64 Command line reference
Shell lesson from Software Carpentry
The Linux Command Line by William Shotts
10 command-line tools for data analysis in Linux by Jason Baker
command line bootcamp
mdn web docs Command Line Crash Course
Ubuntu commandline fore beginners

Install OpenRefine on Mac

Wed, 01 Mar 2017 00:00:00 +0000

Out of Date!

Mac seems to have issues when trying to install OpenRefine–and each version of OS X / macOS has different problems! The official instructions are on the OpenRefine wiki, but this list brings together resources to trouble shoot some common issues during install.

Open Refine needs Java JRE. Go to the Java page, download and install the most up-to-date version for Mac.
- OS X has a few issues with Java, many are documented at Oracle Java on Mac OS X. Older versions of Java are automatically disabled by OS X, so make sure yours is up-to-date. If you try the rest of this install process and Refine still doesn’t work, you may need the legacy Java from Apple.
Download the latest OpenRefine .dmg version (as of this post openrefine-mac-2.7.dmg) from the Releases page.
Install by opening the .dmg and dragging the OpenRefine icon into the Applications folder icon.
- Older versions of OS X might prompt an error message: “Google Refine is damaged and can’t be opened”. This is due to OS X Gatekeeper misidentifying Refine. If you encounter this error, you need to change a setting, in Apple menu > System Preferences > Security & Privacy > General tab > Allow applications downloaded from: > change to Anywhere.
Run by double clicking on the OpenRefine icon in he Applications folder.
- Recent versions of macOS amy give you a warning at this point saying “OpenRefine is blocked from opening because it is not from an identified developer”. Go to System Preferences > Security & Privacy > General tab, and look under Allow apps downloaded from. It should say something about “OpenRefine” was blocked, click the Open Anyway button.
- OpenRefine icon should appear in your dock, and a browser window should automatically open. However, sometimes a browser window does not automatically open. Then manually navigate to http://localhost:3333/ or http://127.0.0.1:3333/ to find the Refine interface.
Shutdown by clicking the OpenRefine icon in the dock and then typing Command-Q.

Handy Linux Apps

Thu, 02 Feb 2017 00:00:00 +0000

Search the repository at Ubuntu Packages to find proper names to install with sudo apt install. If you want to install apps from outside the repositories, download the .deb. Install .deb via commandline using sudo dpkg -i package-name.deb, and remove with sudo dpkg -r package-name (note: the remove package name is not the same as the .deb’s name).

Text Editors

Gedit
Atom
Visual Studio Code
notepadqq, linux notepad++ clone

Utilities

Password database: keepass2
File renamer: pyRenamer, older batch file renamer, but still works. Unique feature allows you to use a csv list to rename a batch of files.
Screencast: vokoscreen, or kazam
File transfer/sync: grsync (GUI rsync), meld (GUI diff and merge).
Backup: deja-dup
Rip DVDs: handbrake (note: the version in the repository is out-of-date, check the official handbrake site for instructions to install the latest)
Flickr uploader: frogr
Ebook manager: Calibre, if you have ebooks, this helps you manage them and connect with reading devices. It also has great built in editing and conversion tools.
Burn SD/USB images: Etcher
Firewall: ufw or gufw
caffeine keeps screen awake during movies
virtualbox VM manager, also if you ever need windows for some reason, get a legal windows VM

Image editing

Image editing: GIMP, gimp (the first time you open it, click “Windows” > “Single window mode” for a more usable experience). If you use raw images, check out Darktable. To play with HDR, luminance-hdr.
Vector drawing: Inkscape or Vectr.

Games

SuperTux Kart, supertuxkart
SuperTux 2, supertux
Endless Sky, endless-sky
mari0
gweled

Handy distros

Volumio, dedicated audio player
RetroPie, dedicated game emulator

Burn SD or USB for Raspberry Pi and more!

Fri, 27 Jan 2017 00:00:00 +0000

MILL-Mini workshop: This session will take you through the Why and How of burning ISO/IMG files to SD or USB for single board computers or installing Linux.

If you are interested in Raspberry Pi, please see more recent Pi intro workshop. I suggest burning images for Pi using Raspberry Pi Imager.

Background

Burning a disk image or bootable USB stick is an important task if you want to use Raspberry Pi or Linux. When you buy a Windows, Chromebook, or Mac computer, the operating system (OS) comes pre-installed. However, if you are interested in running Linux or other open OS, you will most likely have to install it yourself. Traditionally, installation was done by burning a disk image onto a CD or DVD. Since most devices no longer have optical drives, we now create bootable SD or USB instead.

Copying files versus burning an image

A Raspberry Pi doesn’t come with an OS. To get started using your Pi, you need to add a OS to a SD card. You can simply copy NOOBS on to an SD card, and it will magically work! But that is NOT burning a disk image. You have only copied the files on to the SD’s existing file system.

If you want to try any other Pi OS, it will come as an IMG file. Most linux distributions are provided as an ISO file. IMG and ISO are exact binary copies of a storage device. If you simply copy them over to an SD, nothing will happen. The image has to be “burnt”, essentially overwriting the entire media removing the existing file system.

So let’s get an image and burn it to some media!

1. Get an IMG / ISO

For Raspberry Pi and other single board computers, you will usually get an IMG file representing an hard drive image. These are often zipped or in some other archive (.zip or .xz) to decrease the download size. It is not necessary to unzip. For example:

Linux distros are generally released as an ISO, often called a “live cd” or “live usb” meaning it can boot from the install media. For example:

Some linux distros are specifically designed to be portable. You always use them from a USB stick and do not install to a hard drive. For example:

Many specialized utilities are also distributed as images so that they can be run from portable media to make changes to a system without a functioning OS. For example:

GParted Live
DBAN
Anti-virus rescue CDs
- Bitdefender
- AVG

2. Get media

Now you need a micro SD card for Raspberry Pi or a USB stick for linux on a computer. Be sure to get media that is bigger than the image you want to burn!

SD cards: There is a confusing variety of SD cards out there today. Look for one labelled SDHC (SDXC are incompatible with some systems), Class 10 / UHS 3. The numbers aren’t essential, but faster read/write speeds will give better performance. 8GB is standard, but I like to get at least 16GB, since the SD card will be the main hard drive for your Pi. If your computer does not have an SD slot, get a SD reader.
USB drives: Most USB drives will be fine for installing linux. However, if you want to use it as a live USB or portable desktop, look for USB 3 versions with faster read/write speeds if possible.

Optional: Format media. If you have previously used the media, you might want to start with a nice clean disk by re-formatting. If you are on Windows or Mac, get the official SD Formatter. This will enable you to restore SD or USB you have previously burnt, which Windows and Mac are unable to actually read. On linux, the built in formatting utilities are generally sufficient to restore and format media. If necessary use Gparted and format to fat32. However, when using Etcher, formatting before burning is rarely necessary.

3. Get Etcher

Download Etcher for your system, and extract the .zip file (you should get a single .AppImage file on Linux or an .exe on windows).

Why Etcher and not the million other options out there?

Easy to use, it automatically selects the right drives and just works!
Can burn from archived (.zip, .7z, etc) images automatically.
Open source, cross platform, and easy to install.
New and actively developed (most are not).
Fast and efficient, with built in verification of the burn.

4. Burn image

Start up Etcher by clicking on the .AppImage or .exe file. Etcher makes the rest easy!

Plug your media into the computer.
Select the image.
Select the SD/USB (if you have only one plugged in, it will be automatically selected).
Click Flash! (you may need to authenticate at this point, since you are making changes to a drive)
Wait. Wait…
Remove your SD/USB.

Now you can go play with your Raspberry Pi or try out a new Linux distro!

Next, see Boot a Live USB for tips about how to get the USB to boot on your computer.

Gedit tips

Thu, 29 Dec 2016 00:00:00 +0000

Gedit is the default text editor for Gnome and Ubuntu (where its called “Text Editor”). Opened with default options it seems a very basic and not very interesting or useful editor, basically like classic Notepad. However, dig a little deeper, and its a great and powerful text editor.

Here is some tips to set it up with everything you expect from a good code editor:

Default look: open Edit > Preferences.
- On the View tab, check “Display line numbers”, “Highlight Current Line”, and “Highlight matching brackets”.
- On the Editor tab, check “Enable automatic indentation”.
- On Font & Colours tab check “Kate” theme.
- These and other view options are of course totally up to your personal preferences, but the above options are default on most code oriented editors. Note: the line view options can quickly be changed by clicking the Ln, Col information at the bottom of the document.
Spell check: click Tools > “Highlight misspelled words”. Works like a charm! Very handy for markdown or HTML content.
File browser: click View > Side panel, then from the drop down menu above the side panel, select “File Browser”. Toggle with F9. This is very handy when working on a project that is a series of files in a directory.
More plugins: Gedit ships with a series of standard plugins that extend functionality. However, its easy to get a few more from your software repository. On Ubuntu, install some extras with sudo apt-get install gedit-plugins. Then head to Edit > Preferences > Plugins tab to see the new functionality you can enable. Personally, in addition to the standards I enable “Bracket Completion”, “Find in Files”, “Git”, “Join/Split Lines”, “Smart Spaces”, and “Text Size”.

This will get Gedit the functionality you might expect from a code focused editor, while still remaining a lean, easy-to-use, handy app.

Regex tips

Sat, 10 Dec 2016 00:00:00 +0000

Regular Expressions are incredibly handy when processing text files, giving you find & replace super powers. Exactly how they are implemented depends on the tool you are using, both in how you enter the expressions and which functions are available. Learn, practice, and test expressions using RegExr, but keep in mind that it uses JavaScript which doesn’t implement all methods.

Here is list of random expressions I don’t want to forget:

Digital Aladore discussion
find blank lines, (?<![\d\w\S])\n

Use Jekyll Deploy command

Sat, 05 Nov 2016 00:00:00 +0000

jekyll-deploy adds a new jekyll command, accessed on the command line as jekyll deploy. It allows you to add a custom config to quickly deploy the site after build to a different destination.

install

Create a file in root project directory called Gemfile that contains:

source "https://rubygems.org"

gem 'jekyll', '3.2.1'

group :jekyll_plugins do
  gem 'jekyll-deploy'
end

Open a terminal in the project directory and type command bundle install. This will download and install the plugin.

configure

Open the project’s _config.yml. Add deploy command to the end of the config, for example:

# deploy commands 
deploy:
- rsync -r ./ ../../test/

Deploy commands run from the _site directory and can be any command line phrase.

use

First, build your project using jekyll build or jekyll serve. Then type command jekyll deploy. This will activate the command you set in the config.

Download your Gists

Tue, 01 Nov 2016 00:00:00 +0000

I like GitHub Gists a lot. Very handy service! When I start a digital odd job, I like to take notes in Gists, because otherwise I will forget what I did. Gists is a handy memory aid.

However, I realized Gists isn’t the easiest place to find things. So this site was born. Random thoughts and notes-to-self about how to do digital odd jobs. Much of it is half baked or wacky work arounds. but it might help you get something done. At the least it helps me remember and acts as a handy personal reference source.

To start up this blog I wanted to grab all my previous Gists. GitHub API makes this fairly easy for public content. Just choose a language that can download stuff and create files, and you can write up a script.

In my case, all my Gist are a single Markdown file, so its very easy. I just want to get a list of my Gists, download each one, and create a copy of the .md locally. For me it was easiest to modify a Ruby script from blazeeboy. Since my Gist are only a single file, and I wanted them all in single directory to move into a Jekyll project, my version looks like:

#!/usr/bin/env ruby
require 'open-uri'
require 'json'

username = evanwill

# get gists
puts 'Downloading gists list'
gists_str = open("https://api.github.com/users/#{username}/gists").read
gists = JSON.parse gists_str

gists.each_with_index do |gist, index|

	puts "#{index+1}/#{gists.length} Downloading #{gist['url']}"
	gist_str = open(gist['url']).read
	gist = JSON.parse gist_str

	gist["files"].each do |file_name, file_value|
		File.open("#{file_name}", 'w') { |f| f.write file_value['content']}
	end

end

Save the script as getGists.rb in a new directory, then run it by opening the directory in a terminal, and typing ruby getGists.rb. All your public Gists will be downloaded into the directory. If you want each in its own directory, check the original from blazeeboy. If you need to authenticate to get private gists, there are some other options out there.

Update Grub boot order on multi boot system

Wed, 05 Oct 2016 00:00:00 +0000

If you dual boot Windows and Linux, you will probably want to change the default boot order at some point. When logged into Ubuntu, open terminal:

sudo nano /etc/default/grub

Change GRUB_DEFAULT= to the value you want, based on its position on the GRUB menu when your computer starts up (the top OS is 0). You can use the number or use the exact label like GRUB_DEFAULT="Ubuntu 15"

If you want to just use the same OS as was choose last time, set GRUB_DEFAULT=saved and add a line GRUB_SAVEDEFAULT=true

Set GRUB_TIMEOUT= to the number of seconds you want to wait until it boots the default OS.

Exit and save the file.

To update the actual GRUB files, you must now run:

sudo update-grub

done!

Using Jekyll on Windows

Mon, 03 Oct 2016 00:00:00 +0000

Static Gen background links:

lists: StaticGen directory or Static Site Generators
“Why Static Website Generators Are The Next Big Thing” Smashing magazine
Jekyll documentation, intro

Install on Windows 7

Jekyll does not officially support Windows, however it is cross platform (they just don’t officially write windows documentation or check for bugs). There is a Jekyll on Windows page, but it is out of date. Not everything mentioned is required any more.

For more complete install info, and other platforms see go-go gh-pages.

Here are the steps I took to get it running on Windows 7 or 10:

Get Ruby installed:
- Download the suggested stable version of RubyInstaller (Ruby 2.4.X (x64)+).
- Run RubyInstaller, all the defaults should be fine. On the final step, ensure the box to start the MSYS2 DevKit is checked.
- The installer will open a terminal window with options to install MSYS2 DevKit components. Choose option 3, “MSYS2 and MINGW development toolchain”, or simply press ENTER to install all the necessary dependencies. (This installer can be restarted by typing ridk install into a command prompt)
Install Jekyll:
- Open a command prompt, then type gem install jekyll bundler (gem is Ruby’s installer, similar to Python’s pip)
- Done!
Use Jekyll:
- To create a new demo project, open a command prompt, type jekyll new testsite. Jekyll will create a new directory with the given name that includes all the files necessary for a basic project. Once you understand the basics, this step is unnecessary.
- Move to the new directory, type cd testsite
- To start working with it, type jekyll serve. (note: for larger sites you will want to activate incremental build, use the short command jekyll s -i)
- Open web browser and visit http://localhost:4000
- To stop the server, type Ctrl+C in the command prompt.
- While jekyll serve is active, Jekyll watches the project directory and builds a static html site in the directory _site which it serves locally.
- Jekyll watches for any changes you made to the project files and rebuilds the _site as necessary.
- Don’t make changes to the _site directory. It is generated by Jekyll from all the other files.
- Jekyll bundles together a bunch of other helpful development tools, such as SASS for CSS and Liquid for templating pages.
- When you are done developing, the contents of the _site directory will be copied to your host location (for example testweb or github).
- Use a good text editor, since files should be UTF-8, and there can be issues with line endings. I use Visual Studio Code which is handy for navigating the directory structure.

A tour of the project directory

testsite (the top level contains pages that will become top level pages. Edit the overall site settings in _config.yml. Changes to _config.yml are not automatically refreshed during jekyll serve, you have to stop the server and restart.)
- _includes These are modular chunks of your pages which can be called into a page layout by Liquid. For example {% include head.html %}{% end raw %} in a page layout would add the include head.html from the _includes directory.
- _layouts These are basic templates for complete web pages. They are constructed out of includes. The layouts are called by files that have content to add the template structure.
- _posts This directory contains blog posts. We are more likely to use a _collections directory for similar automated treatment of standardized content pages.
- _sass This contains the modular .scss files that will be pulled into your main css for the site. All normal CSS is valid SCSS, but Sass adds many helpful functions and programatic features which are compiled into regular CSS automatically.
- _site This is the built out version of your new static site.
- css Contains the base SCSS file that will be compiled into a single CSS file for the site. You can set all your Sass varibles in one place (if you are using them) and call in your scss partials from _sass by using @import "base";
- You can add other directories as needed for site assets such as images. They will be copied directly to the _site directory.

Example Themes with demos

On gh-pages

Contrary to popular belief, GitHub Pages does allow/support a few plugins. Here is the list: https://pages.github.com/versions/

Add them to your _config.yml, under gems:, not in the the gemfile (since gh-pages ignores that). They can be installed as a bundle using the github-pages gem. Of interest are:

jekyll-seo-tag (GitHub info, usage, basically add {% raw %}{% seo %} to the <head> of your pages, and make sure you have all the metadata in your _config.yml).
jekyll-sitemap (GitHub info, repo).

Use Git locally to collaborate

Fri, 30 Sep 2016 00:00:00 +0000

you can use Git with out a cloud service to simplify collaboration in repository on a shared drive

Set up shared repository

Create a directory on the shared drive to hold the repository, then git init --bare. By convention, bare repos are given the extension .git.

cd /t/
mkdir newRepo.git
cd newRepo.git
git init --bare

The bare repo is not like a normal repository. It does not have copies of the working files visible, only the full git history. Bare repos are only for sharing and collaborating on a repo, not making changes to files. You will see the working files when you clone the bare repo to a normal repository locally.

On your local working drive, create a repository and add the bare repo as remote.

cd /c/
mkdir newRepo 
cd newRepo
git init
git remote add origin /t/newRepo.git
echo "init" > README
git add README
git commit -m "init"
git push -u origin --all

Collaborate

Now some else can make a copy of newRepo from the shared drive /t/.

git clone /t/newRepo.git
cd newRepo

Make some changes or a new branch.

git branch newFeature
git checkout newFeature
echo "new stuff" > newfile.txt
git add newfile.txt
git commit -m "new file"
git push --set-upstream origin newFeature

When you create a new branch you need to git push --set-upstream origin newbranch to set up the new branch in the bare repo.

If you already have newRepo, update by pulling (this should grab any new branches).

cd /c/newRepo
git pull

Now when the new branch is ready for prime time, merge it into master.

cd /c/newRepo
git pull
git checkout master
git merge newFeature
git push

PocketCHIP stuff

Wed, 01 Jun 2016 00:00:00 +0000

The makers of CHIP went out of business as of 2018 and took all their code down. If you want to keep your CHIP running, you have to rely on the community…

Archived stuff

CHIP Community wiki
JF Possibilities, “CHIP is dead.. or is it?” and mirrored the original documentation
yoursunny, How to Flash C.H.I.P Offline.
Thore-Krug, Flash-CHIP

Utility

built in calculator: xcalc

use the built in browser: surf http://google.com (need help, man surf)

get a text browser: sudo apt-get install lynx

get a graphical browser: sudo apt-get install midori

get a paint program: sudo apt-get install mypaint

or: sudo apt-get install grafx2

notetaking: sudo apt-get install xournal

sudo apt-get install evolvotron

From blog

http://blog.nextthing.co/give-your-pocketc-h-i-p-3-new-abilities-software-tutorials-from-the-community/

http://blog.nextthing.co/heres-how-to-host-a-90s-style-doom-lan-party-using-pocketc-h-i-p/

http://blog.nextthing.co/customize-the-hell-out-of-your-pocketc-h-i-p-install-doom-give-it-an-icon-on-the-home-screen/

Fun terminal stuff

Watch star wars in ASCII:

sudo apt-get install telnet

telnet towel.blinkenlights.nl

Get lots of games:

sudo apt-get install bsdgames

bsdgames includes: adventure, arithmetic, atc, backgammon, battlestar, bcd, boggle, caesar, canfield, countmail, cribbage, dab, go-fish, gomoku, hack, hangman, hunt, mille, monop, morse, number, pig, phantasia, pom, ppt, primes, quiz, random, rain, robots, rot13, sail, snake, tetris, trek, wargames, worm, worms, wump, wtf. Play bsdgames by typing the name on the terminal.

sudo apt-get install ninvaders

sudo apt-get install nethack

sudo apt-get install 2048-cli

sudo apt-get install moon-buggy

sudo apt-get install robotfindskitten

rouge-like

http://crawl.develz.org/wordpress/

https://sites.google.com/site/broguegame/

http://www.adom.de/home/downloads.html

Emulators

apt-get install visualboyadvance Then type vba to run it. Vba [rom location] to run the game.

vice: http://www.rift.dk/blog/run-vice-on-your-pocketchip

http://www.lemon64.com/?mainurl=http%3A//www.lemon64.com/games/list.php%3Fpublisher%3D%28Public+Domain%29

To run VICE, type x64 from the terminal. To load a game, tap File then mount a disk, browsing to wherever you put your files.

MEdnafen: https://bbs.nextthing.co/t/mednafen-emulator-configuration-gbc-gba-snes-nes/5027

retroarch: https://github.com/libretro/RetroArch/wiki/Compilation-guide-(Linux)

RetroArch doesn’t require GL. You can compile it without GL/GLES/EGL at the expense of some features (no fancy menus, only RGUI; no shaders). I’m assuming you’ll be able to compile it similarly to how it used to be compiled for rpi (note that opengl/egl/gles are disabled during the ./configure step): https://www.raspberrypi.org/forums/viewtopic.php?t=56070

fceux

Case mods

keyboard face plate, http://www.thingiverse.com/thing:1670579

keyboard, http://www.thingiverse.com/thing:1686723

d-pad, http://www.thingiverse.com/thing:1644139

combine PiGrrl parts?

source files, https://github.com/NextThingCo/PocketCHIP-Mechanical

How to setup Etherpad on a cloud VM

Wed, 01 Jun 2016 00:00:00 +0000

Set up an Ubuntu server, then follow these directions to get an instance of Etherpad up and running! For full documentation check out the Etherpad-lite wiki.

Install dependencies

sudo apt-get install gzip git-core curl python libssl-dev pkg-config build-essential apache2

Install nodejs

curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -
sudo apt-get install -y nodejs

Create a non-root user

add a user to install etherpad, and give it a password:

sudo useradd -m ether-user
sudo passwd ether-user

Get Etherpad

As ether-user (sudo su - ether-user), grab Etherpad from the repository:

git clone https://github.com/ether/etherpad-lite.git

At this point you can start it with the default set up:

cd etherpad-lite
bin/run.sh

It will be running at server.ip.address:9001. You will have to open port 9001 in your network rules to make it available to users. But we should go a bit further to set up a proper database.

Set up DB

Install MySQL sudo apt-get install mysql-server and set a strong password. Start up MySQL with mysql -u root -p then create a database and give your user access with a strong password:

create database etherpadlite;
grant CREATE,ALTER,SELECT,INSERT,UPDATE,DELETE on etherpadlite.* to ether-user@localhost identified by 'goodpassword';
FLUSH PRIVILEGES;
exit

Next, edit the settings.json file for your Etherpad instance to add the correct details about your new database. First, look for the database setting in the file, and comment out the settings for dirtyDB like this:

/* "dbType" : "dirty",
  //the database specific settings
  "dbSettings" : {
                   "filename" : "var/dirty.db"
                 },
*/

Next, uncomment and edit the example given the dirty set up:

  // An Example of MySQL Configuration
   "dbType" : "mysql",
   "dbSettings" : {
                    "user"    : "ether-user",
                    "host"    : "localhost",
                    "password": "goodpassword",
                    "database": "etherpadlite",
                    "charset" : "utf8mb4"
                  },

Set up a proxy to make your URLs nicer

The wiki gives directions for setting up proxies depending on your server. We will use a simple Apache proxy. Go to the Apache2 settings cd /etc/apache2/sites-enabled/. Edit the config file inside, nano default.config. Add the basic proxy described by the wiki:

<VirtualHost *:80>
  LoadModule  proxy_module         /usr/lib/apache2/modules/mod_proxy.so
  LoadModule  proxy_http_module    /usr/lib/apache2/modules/mod_proxy_http.so
  LoadModule  headers_module       /usr/lib/apache2/modules/mod_headers.so
  LoadModule  deflate_module       /usr/lib/apache2/modules/mod_deflate.so
  ProxyVia On
  ProxyRequests Off
  ProxyPass / http://localhost:9001/
  ProxyPassReverse / http://localhost:9001/
  ProxyPreserveHost on
  <Proxy *>
    Options FollowSymLinks MultiViews
    AllowOverride All
    Order allow,deny
    allow from all
  </Proxy>
</VirtualHost>

If you want to have your own index, it is easiest to change ProxyPass / http://localhost:9001/ to one level deeper, such as ProxyPass /pad/ http://localhost:9001/. Then you can use the root level / for your own pages, in the default Apache directory /var/www/html/.

Run with screen

If you just want to get Etherpad up and running without monkeying around with start up, you can run it using screen and detach from your terminal.

screen
sudo su - ether-user
cd etherpad-lite
bin/run.sh

Then type ctrl+a and d. The process will detach from your terminal and continue running. Basic Etherpad is ready!

Add more to Git Bash on Windows

Thu, 26 May 2016 00:00:00 +0000

Git for Windows comes bundled with the “Git Bash” terminal which is incredibly handy for unix-like commands on a windows machine. It is missing a few standard linux utilities, but it is easy to add ones that have a windows binary available.

The basic idea is that C:\Program Files\Git\mingw64\ is your / (“root”) directory according to Git Bash. If you go to that directory, you will find the typical linux root folder structure (bin, etc, lib and so on).

note: depending on how you installed it, the directory might be different. Find it by typing cd / then pwd -W. Alternatively, from the start menu, right click on the Git Bash icon and select open file location. It might be something like C:\Users\name\AppData\Local\Programs\Git, the mingw64 in this directory is your root.

If you are missing a utility, such as wget, track down a binary for windows and copy the files to the corresponding directories. Sometimes the windows binary have funny prefixes, so you should rename the .exe file to the standard name. Since bin is on the PATH, it will be automatically available to Git Bash.

Note: Some interactive utilities, such as python or ipython will not display properly with Git Bash’s mintty terminal. Try invoking them with winpty instead, for example winpty ipython will work nicely. The Git-for-Windows team is focused on supporting Git, not a full UNIX shell environment. If you are interested in more complete package, check my notes on Cygwin or Cmder.

Here are some utilities to add:

Wget

Download the latest wget binary for windows from eternallybored (they are available as a zip with documentation, or just an exe)
If you downloaded the zip, extract all (if windows built in zip utility gives an error, use 7-zip).
Rename the file wget64.exe to wget.exe if necessary.
Move wget.exe to your Git\mingw64\bin\.

Note: I have noticed some bugs when using Wget on Git Bash to create WARC files. For more complex use of Wget, consider Cygwin instead.

Hugo

Hugo static site generator can be downloaded as a binary and does not have a installer. Dropping it into your bin easily adds it to your Git Bash path.

Download the Windows 64-bit version from the releases page.
Unzip, then copy hugo.exe into your Git\mingw64\bin directory.

Xpdf

Xpdf is a handy utility for manipulating PDF files.

Download the windows version “Xpdf tools”.
Extract zip.
Copy the contents of xpdf-tools-win-4.00\bin64\ into your Git\mingw64\bin\.
Check the docs to get started with tools such as pdftotext and pdftopng.

ExifTool

ExifTool is a utility to read and write embedded metadata in image files.

Download the “stand-alone Windows Executable” from the ExifTool page (this will be a .zip file e.g. “exiftool-11.20.zip”).
Unzip the downloaded file.
Inside you will find a file named exiftool(-k).exe. Rename it to exiftool.exe.
Copy exiftool.exe into your Git\mingw64\bin directory.

make

Go to ezwinports.
Download make-4.1-2-without-guile-w32-bin.zip (get the version without guile).
Extract zip.
Copy the contents to your Git\mingw64\ merging the folders, but do NOT overwrite/replace any existing files.

Nano

As of 2018, recent versions of Git Bash include Nano, so this is unnecessary!

Download the Nano binary from Nano win32-support page. You just need the .exe file, which is named nano-git-0d9a7347243.exe (as of this writing).
Rename the file to nano.exe, and copy to the mingw64\bin directory.
This version of Nano will not work with Git Bash alone, but can be invoked using winpty, for example, winpty nano test.txt.

Writing in Markdown

Fri, 20 May 2016 00:00:00 +0000

Markdown is a quick and simple way to write formatted documents in plaintext. It is intuitive and easy to read, but is designed to convert to HTML for easy web publishing. Writing in plaintext rather than proprietary formats such as Word has many advantages:

simple - don’t waste time in complex and frustrating formatting in Word that will be deleted by publishers anyway. This keeps your distractions to a minimum.
sharable - any device can open and read plaintext, no special software needed, unlike Word.
preservable - markdown is plaintext which is easy for digital preservation, unlike Word.
version controllable - you can use the full power of Git or other version control systems, unlike Word.

GitHub Markdown

Markdown was originally developed by Daring Fireball. However, other “flavors” of md have developed. The most popular is GitHub markdown flavor, since it can be used anywhere on GitHub.

Here are some tutorials:

Markdown editors

Markdown is plain text, it’s sort of like source code for your document. You can write it in any application that supports plain text, i.e. a text editor. When working with code you should have a good text editor.

For basic editing, Windows Notepad++, Mac TextEdit, or Linux Gedit are sufficient (note: you can use Windows Notepad, but it does not handle UTF-8 encoding or UNIX line endings that are standard for most cross platform applications). However, more advanced code editors often have plugins or features to make it easier with previews of your document. Checkout Visual Studio Code (use shortcut Ctrl+Shift+V to preview) or Atom.

If you want a more fully featured visual writing environment, there are also applications devoted specifically to writing in Markdown. These apps will have previews, GUI controls, and automatic export options.

Web-based:

Dillinger
StackEdit
HackMD (real time collaboration)

Stand alone apps:

Remarkable - code / preview split screen.
Typora - live preview writing focused slick new app.

Rendering Markdown

Markdown is human readable as plain text–it is intuitive like writing an email. However, it is designed to render as HTML. Markdown compilers are built in to many applications (like the editors above) to generate formatted versions such as HTML, .docx, or PDF.

One option is Pandoc, a command line utility to translate between many formats and generate new output versions, such as PDFs (via LaTex). Check my note Pandoc for PDFs or the Sustainable Authorship in Plain Text tutorial.

UnLenovo

Tue, 15 Mar 2016 00:00:00 +0000

Things to do when you get a new lenovo computer

Uninstall everything from Lenovo! Its all sketchy bloatware and you don’t need any of it at all. Seriously. Win 10 has everything you need.
Navigate over to C:\Program Files (x86)\Lenovo and look in all the directories there and click the unistall.exe files! Cause guess what? Lenovo put some even sleezer things in there that you can’t uninstall via normal means. Then delete all the directories.
Just to be sure–You dont need anything in this ^ directory! Customer Engagement Service (CCSDK) and others log your activities and send information to weird unsecure web locations! Yuck!
Go into control panel, device manager, and update drivers for wifi, audio, video, intel power management.
okay, it should be better now!

Pi Py Poetry, Pi Day Remix

Mon, 14 Mar 2016 00:00:00 +0000

Its 3:14 on 3/14 (2016)! Time for some Vandal Poem of the Day (VPOD) poetry remix using OpenRefine and Python.

First, I grabbed all of the poems from VPOD.

Then I parsed it all using OpenRefine.

I Refined it down to a table with these columns: title, author, poem, words, length. Each row represents one line from a poem in the corpus. The line is in the ‘poem’ column. The ‘words’ column is a count of words in the line, created in OpenRefine with new column based on ‘poem’ > length(value.ngram(1)). The ‘length’ column is character count of the line, created in OpenRefine with new column based on ‘poem’ > length(value). The ‘title’ and ‘length’ are not used in this example. To simplify I sorted on ‘words’ and used only 1 through 9, since these will match the Pi digits.

Then I went into Jupyter Notebook to get started generating poems with Python:

# import stuff 
import pandas as pd
import random

# create pandas dataframe from the poem CSV
poemDF = pd.read_csv('VPODpoemLines1-9.csv')
# create list with digits of Pi
piDigits = [3, 1, 4, 1, 5, 9, 2, 6, 5, 3, 5, 8, 9, 7, 9, 3, 2, 3, 8, 4, 6, 2, 6, 4, 3, 3, 8, 3, 2, 7, 9, 5, 0, 2, 8, 8, 4, 1, 9, 7, 1, 6, 9, 3, 9, 9, 3, 7, 5, 1, 0, 5, 8, 2, 0, 9, 7, 4, 9, 4, 4]

This gave me a dataframe with 3344 rows (i.e. lines of poetry. I should have brought it down to 3,141!), representing lines of from 155 poems by 18 VPOD authors. Find that info with:

len(poemDF['author'].unique())

len(poemDF['title'].unique())

Okay, now we can make some new remix Pi Py Poems:

# create a random poem based on digits of Pi
# first decide number of lines in the poem, 
# I based this on the approximate range observed in the corpus
poemLength = random.randrange(5,60)

# create a poem of that random length
# the digits of Pi are the number of words per line
for i in range(poemLength):
    if piDigits[i] == 0:
        print("\n")
    else: 
        newLine = poemDF[poemDF['words'] == piDigits[i]].sample().values[0][2]
        print(newLine)

Some Pi Py Poems:

Then more lights
undefeated,
as a train speeding
prose
Her curling makes me shiver
how to open the airway of a fellow passenger
it's lying
and I marry you each day.
Because my heart could not
what they were
like the residue of beets
I can do so only for a moment.
barrel, she broke him. She gave the worst smile:
has approved all your plans, so you
I could never deny it, or disown my desire
No, he said.
    the feet
black bird hunched
But it's a reassuring logic that rivers freeze
Now I remember something

Because of breathing.
*
and the snow outside
*
and burning whatever is there.
We'd rut a ditch by a river in nights
everyone with
violet towers. We scour the earth
If you write “ironic detachment”
we couldn't hear
breaker of others' bones, parachuter,
The one called Chukwu? Just one word: God.
with a plastic teapot for my sister's tea party.
a child's Christmas miracle that I let
Can I come back up now? asks the chair
my worldly nature
Or burn
are spelling someone's
but more and more it seems it won't.
It looks like smoke
the mustached man with his shoulders
to disappear
when it is opened or closed.
affix their wintry incubus,
our scarves, shrink
and life, less
Enter a room as though it is strange.
holding a jar
Glass skins
the wonder of them, good or bad,
I know you… the skin graft on your cheek,
to dream up only this.


emitted light
a new home, maybe you'll search and petition
But it's a reassuring logic that rivers freeze
like some plastic beads
Yes.
maybe you no longer haul those wounds with you

off the closets
they're
and trained it stranger
Delivered
some type of old-fashioned candy
Think of the flesh on an angel's hips, pinched
in our
to do-- an aperture, a slur-- 
And blown between adjacent bridges
the giant translucent
light. And after the light,
I eat what's put in front of me,
my fear of needles, my cravings for salty food,
with empty wine jugs at our heels.
of the space movie we all stood in line
she came over
in our
a phoenix preening
    to empty your bright dress onto the floor,
fringe on brown pain
Even if I have convinced you,
fair, alone.
a world flashes by, and opposite,
certain spells to protect
in the morning
of not remembering
with a dark beer or hand out leaflets
Washed in salt
The current
then finger, forearm and elbow. Curl yourself
No I am not afraid of the eye chart:
there are experts among us


The current
made a circus of it. Asthma, weeping, elephants,
and through my reading to a deafening applause.
and hope it's clear.
alive
to own anything with and without blood. Cash rules
On the shore, the line of unannointed
undefeated,
Consume me and find me worthy.
You knock and knock at a door, it won't
to live. Mother,
feel silly in your grief. And still you'll sit
by the heat in the tips of her fingers.
even on holidays.
the silhouette of a single red-mouthed bell;

Good Fun!

Add Authors

Next I wanted to add authors to the Pi Py Poems. I modified the poem-creating loop like this:

poemLength = random.randrange(5,60)
author = 'Pi Py Poetry'
# create a poem of that random length
# the digits of Pi are the number of words per line
for i in range(poemLength):
    if piDigits[i] == 0:
        print("\n")
    else: 
        newLine = poemDF[poemDF['words'] == piDigits[i]].sample()
        author += ', ' + newLine.values[0][1]
        line = newLine.values[0][2]
        print(line)
print('\nBy '+author)

The New poems result in a crazy looking author list, but you can track down the original lines by author (potentially). More examples are below.

More Pi Py Poetry from VPOD

in another room
rereading
in magic cochineal pants.
Shifting
snaps free. Sunflash, stereo store,
The sage and tomato plants tanning on the veranda.
The current
    and pretend we are lying there
which said he was born
I once read
and you're never getting out.
places to disappear but it was fooling itself,
they are stuffed in every corner of a house
all this silence and emptiness and rust
for the certainty of home, for mills and reservoirs
Every question, conversation
beneath the
I do remember
I close the shop at six. Welcome wind,
this time of year
to step from the wrought-iron compartment,
She's alive
only a dream is so lucky.
dressed like a rabbit
think they're alone.
Resurrection and crucifixion.
Whatever you say sounds better with your thigh
I float bestowing
without knowing
I'll admit, she's earned her orchestra seats
    they attempt to pull the survivor from its flood
there remains more to learn.


how often.
Not like what they told us at all
in a field, two and a half days.
But I'm dry now,
rereading
The pay toilets where we sat without paper. Rain.
There is a dawn between my legs,
embarrassing
covers from romance novels, their heroines
And a call to joy-- a landscape, a face-- 
then thrown away
to love the dark. I love it so much
you feel sick in ways you thought not possible.
sip false peace
who are dead now laugh and laugh.
grow especially talkative at night,
senseless.

By Pi Py Poetry, Laura Kasischke, C.D. Wright, Natalie Diaz, Laura Kasischke, James Arthur, Roger Reeves, Kerry James Evans, Natalie Diaz, Matthew Zapruder, Matthew Zapruder, Mark Bibbins, Dean Young, Valzhyna Mort, Laura Kasischke, James Arthur, Dean Young, Michael Dickman, Matthew Zapruder, C.D. Wright, Brenda Shaughnessy, C.D. Wright, Michael Dickman, Brenda Shaughnessy, Matthew Zapruder, Brenda Shaughnessy, Chris Abani, Mark Bibbins, Matthew Zapruder, Matthew Zapruder, Erin Belieu, Sherwin Bitsui, Lisa Olstein, Brenda Shaughnessy, Michael Dickman, Lisa Olstein, Kerry James Evans, C.D. Wright, C.D. Wright, Natalie Diaz, Brenda Shaughnessy, Erin Belieu, Sarah Lindsay, Valzhyna Mort, Lisa Olstein, Chris Abani, Matthew Zapruder, Laura Kasischke, C.D. Wright, Brenda Shaughnessy

but there's still
being
Tunisia of desert silence
6
tell us his tired children
Coupling beneath my feet. I want the body's burden,
could theoretically
should I wish to review them.
contain you, it refused sympathy,
Cigarette in hand.
while you sleep? Sleeping is
trapped in the bill of a circling bird,
and I don't bruise easily, that I am yours
so two sets of passengers come eye-to-eye
because your hemisphere has rolled away from the sun,
turns into breath
and senses
a phoenix preening
it's my leg lifted over my man's body.
Consumers will pay more
belled purple cluster, I'd rather plump-girl-
books die
As always I can do nothing.
have forgotten fear, apparently
a phoenix preening
traveling toward us
I stomp down a refinery, trail a scat
Not your name
if stars
a blood orange swelling like a breast-- 
for slavers. In the lake, red dye bubbles up
    to be mauled by wolves


heaven, like
that will eventually like me into the earth
wiping his greasy hands on a greasier rag,
a bright green snake
created
and every house carries a legend of a captain
In the oven, something breathing. Rising. Melting.
hair.
my brother's leg destroyed after detonation;
of a neighborhood pool floats up on the heat.
from the '70s
a rent check or explains anything to one's family.
My fear of uniforms is an old habit, comfortable.
960 antitank mines

By Py Pi Poem, Michael Dickman, Laura Kasischke, Matthew Zapruder, Natalie Diaz, Matthew Zapruder, Roger Reeves, Matthew Zapruder, Lisa Olstein, Lisa Olstein, Laura Kasischke, Lisa Olstein, Erin Belieu, Brenda Shaughnessy, James Arthur, James Arthur, Brenda Shaughnessy, Valzhyna Mort, Natalie Diaz, Valzhyna Mort, Lisa Olstein, Erin Belieu, Valzhyna Mort, Matthew Zapruder, Sarah Lindsay, Natalie Diaz, Laura Kasischke, James Arthur, Michael Dickman, James Arthur, Natalie Diaz, Chris Abani, Sherwin Bitsui, Laura Kasischke, Matthew Zapruder, Dean Young, Lisa Olstein, Michael Dickman, Valzhyna Mort, Laura Kasischke, Laura Kasischke, Kerry James Evans, Erin Belieu, Matthew Zapruder, Erin Belieu, Chris Abani, Kerry James Evans

in the home.
waistless,
be there, get it?
•
Already you're in the air
even lightning can go wrong but when the smoke
having pressed
not blue-veined, but full of flesh,
that weren't fenced or watered.
a phoenix preening
only in the exit wounds,
to sip at the lachrymal glands of chaos,
floating through the candled dark in shiny black slacks
then I wake up. Did you know
a Vespa in the second person, want it to
pleasuring your feet,
You know
your own shadow,
register key. And yet, how fine it feels,
my dreams mostly water.
There are bulls between my legs,
They stand
white votives shivered in red glass
on a report card.
into the ocean
Smoking a cigarette
that he would give his life for you.
into the sky
diamond after
then left in the shade to multiply.
with a hollow pen, how to wrestle an alligator,
some type of old-fashioned candy


another ambulance
The one called Chukwu? Just one word: God.
I've never bothered with the names of flowers,
that's the mortal art.
*
a Vespa in the second person, want it to
and without your permission. O dummies dancing
reading
Heretics banged at the double door.

By Pi Py Poetry, Brenda Shaughnessy, Brenda Shaughnessy, Brenda Shaughnessy, Michael Dickman, Dean Young, Dean Young, Matthew Zapruder, Roger Reeves, C.D. Wright, Natalie Diaz, Brenda Shaughnessy, Dean Young, Michael Dickman, Lisa Olstein, Mark Bibbins, Erin Belieu, Michael Dickman, Brenda Shaughnessy, Erin Belieu, Dean Young, Natalie Diaz, Valzhyna Mort, Natalie Diaz, Valzhyna Mort, Michael Dickman, Michael Dickman, Chris Abani, Michael Dickman, Michael Dickman, Laura Kasischke, Lisa Olstein, Laura Kasischke, Natalie Diaz, Chris Abani, Erin Belieu, James Arthur, Michael Dickman, Mark Bibbins, Roger Reeves, C.D. Wright, Natalie Diaz

We said despair
Both
Or am I alone?
Imperceptible
that veins blued through it
Blue gourds glow and rattle like a two-man band:
believe, too.
puffballs tenderly bunched in my armpits.
all-y, all-y, all come free
with exacting coordinates
I was never your Intended,
whose ruby shoes throb on shelves in closets,
bark the tips of its leaves with cracked amber-- 
are gone now; we never held them
to the left of the graveyard, where the trees
The throb lingering
harmlessly decompose

in Peru 14%
    Noisier
Maybe they used to
dead
from the one who loves,
walk around in the yard not making a plan.
waist-high, elbow-deep-- 
particular I didn't come to know:
I'd thank it for making
it is quiet
is good enough advice. [Move
Four more angels to drag through the streets
A foot chalking a valley black and bright beneath
has approved all your plans, so you
For years I could count on waking to plenty,
eat another apple
but forgot
who must sense
to sip at the lachrymal glands of chaos,
aglow with determination, pursuing
of shame dissolving in my throat;
Can't matter
it's garbage night. The moon agrees,
of plastic pop-up targets
The fire glowing
The final arbitration.
But the plain ones are patient, aren't they?
And follows itself
The foghorns
and quickly became symbolic, bound to occur
    they attempt to pull the survivor from its flood
in the grass the bodies


believe, too.
The first cell felt no call to divide.
since Cousin Bobby told her about a comma
Love. Hunger. Other alchemies.
*
were starting to cover the land and someday everyone
was a thing of incomparable curvaceous length;
today
and i am on my knees
to love the dark. I love it so much
turns into breath
Like a sharpshooter who knows to pull the trigger
of the space movie we all stood in line
    on both sides,
their mother invited next to the fire,
a nursery for new stars.

By Pi Py Poetry, Matthew Zapruder, C.D. Wright, Michael Dickman, Michael Dickman, Brenda Shaughnessy, C.D. Wright, Lisa Olstein, Erin Belieu, Mark Bibbins, Matthew Zapruder, Mark Bibbins, Roger Reeves, Roger Reeves, Laura Kasischke, Sarah Lindsay, Natalie Diaz, Matthew Zapruder, Brenda Shaughnessy, Dean Young, Sarah Lindsay, Erin Belieu, Brenda Shaughnessy, Mark Bibbins, Kerry James Evans, Laura Kasischke, Chris Abani, Erin Belieu, Brenda Shaughnessy, Michael Dickman, Sarah Lindsay, Sherwin Bitsui, Natalie Diaz, Brenda Shaughnessy, Sarah Lindsay, Sarah Lindsay, Laura Kasischke, Michael Dickman, Matthew Zapruder, Sarah Lindsay, Michael Dickman, Valzhyna Mort, Lisa Olstein, Brenda Shaughnessy, Lisa Olstein, Matthew Zapruder, Kerry James Evans, Erin Belieu, C.D. Wright

And Remember: Read real poems daily at Vandal Poem of the Day

Open image editing

Tue, 01 Mar 2016 00:00:00 +0000

Need to edit some photos? Need an alternative to absurdly expensive and user unfriendly commercial image software (i.e. photoshop)?

Luckily, the free software community has you covered. Here are some notes about some options. If you need more tools and processing workflows, check out the resources at PIXLS.US.

Raster: GIMP

An essential utility for every computer, GIMP (GNU Image Manipulation Program) is a complete raster graphics editing software package. Download it or install from your software repository. When you first open the application, its in the difficult to use multiple window mode. Personally, the first thing I do is click the “Window” menu and select “Single-Window Mode”. This docks the various tool bars into a single window (much like modern versions of photoshop). To me it seems much easier to use in this mode, although many power users prefer the other mode.

Also check out Darktable, the “workflow application and raw developer”, kind of like Adobe Bridge.

Other handy tools

Inkscape is a decent free and open vector graphics application.
ImageMagick command line image processing / creation tool. Handy for batch processes, image type conversions, or editing with another programming language. It comes installed on most linux distros, so you can just open a terminal and type something like convert test.jpeg test.png to start using it. Or you might resize a whole directory of images like convert '*.jpg[300x300]' squares%03d.png and then make them into a GIF like convert *.png images.gif.
G’MIC image processing language / library / plugin.
Hugin Panorama stitching utility. I misuse it to create stitch nonsense together…
Luminance HDR HDR utility. I use it to create glitchy images from arbitrary sets of images…
ExifTool image metadata utility, the only reliable way to read and write photo file embedded metadata. Or you can try Exiv2.
Ifranview if you are on Windows, this is a handy viewer with very good batch utilities built in.

Download metadata from flickr

Tue, 01 Mar 2016 00:00:00 +0000

You can bulk download stuff from flickr very easily now (which is a great feature), by going to Camera Roll, selecting things, and clicking download. Or you can bulk download your sets/albums now as well by visiting the set/album and clicking the download button. This is handy, but it doesn’t export any metadata you added to the photos in the system.

Unfortunately, I found none of the apps are very stable. The most recent and least broken apps are:

Flickr Downloadr, https://flickrdownloadr.com/
FlickrMetadataSynchr, http://flickrmetadatasynchr.codeplex.com/

Linux has one good option, but set up takes forever if you have lots of photos,

picty, https://github.com/spillz/picty

Exiftool

You want exiftool, nothing else consistently works to look at metadata.

Download the complete version, http://www.sno.phy.queensu.ca/~phil/exiftool/
extract the files
in the extracted directory, use ./exiftool picture.jpg to look at exif data of the pic without installing.
use ./exiftool -json=picture.jpg.json picture.jpg to write json metadata to the embedded exif data of the image (you get the json or xmp file from other applications).

Disable Autoplay

Tue, 01 Mar 2016 00:00:00 +0000

Autoplay video and audio ads and content are amazingly annoying. Anyone who lived through the era of popups knows ad creators don’t care about user experience, but I still don’t understand why they think aggressive, annoying, and abusive additions to a web page will actually help sell stuff. Maybe they are driven by analytics numbers showing users interacting with the ads–unfortunately in most cases it’s users trying to figure out how to block, disable, and dismiss the intrusion. Personally, I hate autoplay content just as much as ads, but some website seem convinced you want video and you want it to play immediately, and you don’t want to be able turn it off or hide it on your own. I mean, what do you know anyway, your just the user??

In any case, ads that run using Flash or other plugins can and should be blocked simply by having all plugins ask before running. Many autoplay ads will be blocked by an adblocker, such as uBlock Origin. However, HTML5 autoplay content videos / audio are a bit different.

Here is some options to block them:

Firefox

add Flashstopper plugin, which despite its name blocks HTML5 autoplay as well as Flash.
If you want to go further, Firefox has a relatively new and hidden setting to prevent most from playing as you load the page, although its not perfect. In your Firefox address bar, type about:config to bring up the manual configurations. If you haven’t used it before, you will have to dismiss a warning about how its dangerous to modify things. Yep. Now search for autoplay using the box at the top. You should find a preference called media.autoplay.enabled with the value true. Right click on the value and choose Toggle to change it to false. Close the config, and restart Firefox. Most HTML5 videos set to autoplay should now be blocked! The only problem is that a lot of sites use video autoplay for content you actually want to see and haven’t implemented their videos properly to allow the user to start them manually. For example, video on Twitter doesn’t seem to work with autoplay disabled. This is either bad developing, not realizing that there is use cases where a video needs to be started manually, or just intentional anti-feature to force you to watch auto play spam if you want to watch their legitimate content.

Chromium / Chrome

add Disable HTML5 Autoplay plugin. Unforturnately, this plugin is not actively developed (check news at the repository). There is some hints that the functionality will move to uBlock Origin.
be sure to enable tab muting (available by default in Firefox): paste chrome://flags/#enable-tab-audio-muting into the address bar. Click Enable to toggle the feature on. Now you can click on the little speaker icon on any tab to mute the audio.

Text Analysis Notes

Thu, 18 Feb 2016 00:00:00 +0000

Collection of Text Analysis resources from various workshops and presentations.

Text Analysis?

= a messy term that encompasses many interconnected processes such as text data collection, cleaning, parsing, summary, visualization. Also known as text mining.

Examples: Text Annotation, Natural Language Processing, Sentiment Analysis

Explore a Text

remember: Your Data == Text

Juxta (digital text collation. NINES.)

Juxtacommons (online version, great for sharing)
desktop (legacy desktop version)
My Digital Aladore example

Voyant Tools (suite of online text visualization tools. Stéfan Sinclair & Geoffrey Rockwell)

Wordseer (suite of tools run on local server)

TokenX (“a text visualization, analysis, and play tool”. Brian Pytlik Zillig, U of Nebraska Libraries)

TokenX web app

WordTree (Jason Davies)

NLP

Stanford NLP Group (a library of Java apps, e.g. named entity tagging demo
Open Calais (API trained on web and newspaper text)
Watson Natural Language Understanding (API trained on web content)

Explore a Corpus

Topic Modeling

Megan R. Brett, “Topic Modeling: A Basic Introduction” (2012)
MALLET (MAchine Learning for LanguagE Toolkit) (Programming Historian MALLET lesson)
Topic Modeling Tool (simple visual way to use part of MALLET)
- example classroom projects: Sherlock Holmes’s London prep and analysis; Posner basic strategies and tmt_get_started.

Overview Docs (online tool designed for journalists to sort through huge data sets)

Overview web app
Good help docs

Jigsaw (“Visual Analytics for Exploring and Understanding Document Collections”)

Concordancers

AntConc (lots of software and publications from Laurence Anthony)
CasualConc (R package)
TextSTAT

Explore a Huge Corpus

Think about Big Data 3 V’s (volume, variety and velocity).

Ngrams

Bookworm
Google Books Ngram Viewer (see TED talk)

Hathi Trust Research Center Portal (big text data)

Twitter analysis

Healey & Ramaswamy, “Visualizing Twitter Sentiment” (2013)
Clement Levallois, “Umigon: sentiment analysis on Tweets based on terms lists and heuristics” (2013)
now everyone is doing it! Twitter interactive

Explore Text with Programming

Python

Distribution: Anaconda (get Python 3, 64-bit)
Tool Kit: NLTK (install via conda install nltk or pip install nltk)
Learn: NLTK Book (Steven Bird, Ewan Klein, and Edward Loperm, designed to teach text analysis)

Distribution: R
IDE: RStudio
Learn: Text analysis with R for students of literature, Matthew L Jockers (New York : Springer-Verlag, 2014).

Bash Shell

The command line has lots of great functions for manipulating text files!
Programming Historian, Introduction to the Bash Commandline
SWC, Unix Shell

Tool Catalogs / Directories

DIRT (Digital Research Tools)
TAPoR3 (“Discover research tools for studying text”)
DH Commons (good place to find example projects)

Caution: Due to the nature of academic funding cycles, there is a lot of dead tools/projects out there, and a lot of tutorials for dead tools. Many of these tools still work, but without active maintenance they may not for long. Due to the technical and statistical nature of these tools, descriptions of what they do and how they work may be Difficult reading… Don’t be intimidated! However, it is good to be aware of the academic literature explaining the tools and algorithms since you may need to cite them to validate the techniques in your own work.

Helpful DH Resources

Overheard: stopwords, “to be or not to be”

wget notes

Fri, 15 Jan 2016 00:00:00 +0000

Wget is a handy commandline utility for grabbing stuff off the web. It is built into most UNIX systems. If you are on windows, install Git for Windows which comes packaged with Git Bash, a very handy UNIX terminal emulator (note: check if Git Bash has wget installed by typing wget --version. If it is not installed, check this note). Other windows options are Cygwin, MobaXterm, or Bash on Windows Subsystem.

See all the commands by typing wget --help in your terminal or visiting the man page.

Download files from a list

Create a plain text list of urls you want to download, one per line. Use the -i option to pass that list to wget.

wget -i download-file-list.txt

Download all files of a type from a domain

Use the -r (recursive) and -A (accept) options. For example, download all PDFs:

wget -r -A.pdf http://page-with-pdfs.com/

limit to file type: --accept html

Check dead links

wget --spider -o test.log -e robots=off -w 5 --random-wait --mirror --page-requisites --no-parent http://www.example.com

Fully mirror a site

Download everything you need to have an exact copy of a full site locally:

wget --mirror --convert-links --adjust-extension --page-requisites --wait=2 --random-wait http://example.org

If you only want files from a specific set of domains use --domains=LIST with a comma separated list. If you just want to crawl part of a domain use --no-parent flag. If the site has redirects to send you to an authorative version of a page, you might want to use --trust-server-names, which will name the files based on the file URL rather than the original URL (before redirects).

WARC web archive options

To create a WARC for any wget crawl, simply add the --warc-file="filename" flag to the command. Wget will harvest the site assets as normal, but additionally create a WARC compressed as a .gz file. For larger sites it’s a good idea to add --warc-max-size=1G to limit the max size of each WARC so they don’t get too big.

Combined with a mirror, it might look like:

wget --mirror --wait=2 --random-wait --warc-file="example.org-archive" http://example.org

If necessary you can also send a different user agent if the server refuses to give content to wget’s default one, like --user-agent=Mozilla. Occasionally it might be necessary to ignore robot.txt for archival purposes, add -e robots=off to the command.

If you want to view your WARC later, use Webrecorder Player.

Download a file and change the filename

Let say you have an API with funny url that gives you an image. Wget might save it with some bizarre filename. You can change the file name with the --output-document option (-O). For example:

wget -O new.jpg "http://example.org?CISOROOT=HJCCC&CISOPTR=9&action=2&DMSCALE=100&DMWIDTH=9000&DMHEIGHT=9000&DMX=0&DMY=0&DMTEXT=&DMROTATE=0"

Handy Raspberry Pi Commands

Tue, 05 Jan 2016 00:00:00 +0000

cheat sheet of tips mainly for headless RPI

First time set up

default password for “pi” is “raspberry”

First time set up: sudo raspi-config

update everything: sudo apt-get update && sudo apt-get upgrade

General

start GUI: startx

shut down: sudo shutdown -h now

SSH

ssh into the pi: ssh [email protected]

copy files to pi: scp filename [email protected]:~/destination/foldername/

copy files From pi: scp [email protected]:~/source/file /destination/directory/

install VNC: https://www.raspberrypi.org/documentation/remote-access/vnc/

start a vnc server: vncserver :0 -geometry 800x600 -depth 16

stop: vncserver -kill :0

Networking

check network config and get ip address: ifconfig

edit config: sudo nano /etc/network/interfaces

Startup

edit startup: sudo nano /etc/rc.local

Mount USB drive

plug in the usb drive
find what the drive is called with ls -l /dev/disk/by-uuid/, normally it will be /sda1
create a directory to connect to the drive, i.e. a mount point: sudo mkdir /media/usb
make the main user, i.e. pi, the owner of the mount point: sudo chown -R pi:pi /media/usb
mount the drive: sudo mount /dev/sda1 /media/usb -o uid=pi,gid=pi (replace /dev/sda1 with your actual drive name if it is not the usual setup)

Use a HDMI to VGA adapter

If you are trying to use an HDMI to VGA adapter, the pi will not detect the HDMI screen and will default to using composite video only. You need to edit your /boot/config.txt file on the pi’s SD card to manually tell it to use HDMI. See eLinux RPiconfig for full info about config.txt.

Put your pi’s SD card in a computer’s card reader.
Find config.txt in the boot directory (which will likely be the only thing you can see from your computer).
Open /boot/config.txt in an editor.
Uncomment the line hdmi_safe=1
save

OpenGL

Try it out with a few OpenGL games:

sudo apt-get install neverball

sudo apt-get install oolite

Template note

Fri, 01 Jan 2016 00:00:00 +0000

Basic template for _drafts notes.

My standard demo list:

dog
cat
muffins