Migrate from Anaconda to Miniforge

Posted on by

Continuum seem intent on shooting themselves in the foot with their new license and purchase terms so my employer is moving away from Anaconda with the forge default to Miniforge with. the forge conda-forge.

Now you might think there was a spot where we could change “default” to “conda-forge”, our work would be done and we could celebrate with magic cupcakes. Things aren’t that easy. To make sure it is done right we need to go through all our python projects and export a list of packages in the environment1. To make the code simpler we put all the projects into a folder called “SHIFT” in your home folder. They don’t have to stay there, they can be moved back to wherever they came from once we finish.

Then we remove Anaconda, making sure there are no remnants remaining. The penultimate task is downloading miniforge and installing it. Finally we recreate the environments we had in Anaconda using those files Anaconda wrote.

anaconda_migrate.py
#!/bin/zsh
#
# Anaconda to miniforge migration
#
# we are assuming that the projects using anaconda are all lined up in the one
# directory ~/SHIFT/
for i in ~/SHIFT/*; do
    if [ -d "$i" ]; then
        cd "$i" || continue
        HERE=$(basename "$i")
        # we are assuming the conda env name is the same as the project directory name
        conda activate "$HERE"
        conda env export > environment.yml
        # some authors of deprecate instructions point out `conda env export`
        # outputs lines like:
        #     some_package=1.2.3=py38h1234567_0
        # which conda cannot parse, so we need to strip off the build specifier
        # why conda cannot parse it's own output is beyond me.
        sed -i '.bak' 's#\(^.*=.*\)=#\1#' environment.yml
        cd - || continue
    fi
done
# Remove anaconda and all it's fluff
conda init --reverse --all # undo conda init changes to shell config files
conda install anaconda-clean 
anaconda-clean --yes # remove all anaconda-related files
# let's be sure
rm -rf "~/anaconda*/"
rm -rf "~/.anaconda*/"
rm -rf "~/Library/Receipts/io.continuum*"
# Install miniforge
curl -LsS \
https://github.com/conda-forge/miniforge/releases/download/25.11.0-1/Miniforge3-25.11.0-1-MacOSX-arm64.sh \
> Miniforge3-25.11.0-1-MacOSX-arm64.sh
chmod +x Miniforge3-25.11.0-1-MacOSX-arm64.sh
./Miniforge3-25.11.0-1-MacOSX-arm64.sh -b -p $HOME/miniforge3
rm Miniforge3-25.11.0-1-MacOSX-arm64.sh
for i in ~/SHIFT/*; do
    if [ -d "$i" ]; then
        cd "$i" || continue
        conda env create -f environment.yml
        cd
    fi
done

  1. Just so you have an idea of how bad conda is notice that export can create lines that can’t be parsed in an environment package list. We have a piece of software that cannot parse it’s own output. ↩︎

Help with regular expressions

Posted on by

I recently re-discovered stream-editor which is a neat tool for working on regular expressions. It even allows you to do some chaining of tools.

It runs on some sort of Linux box so there are one or two gotchas for the Mac version of some tools. I may document a method to install it on a Mac. The source is on GitHub

I may even host it on the Apple Silicon Mac Mini sitting unloved in my box room.

Hunting Again

Posted on by

My 12 months at the Commonwealth Bank will soon be over. Therefore I need to find a new position. The 12 months at CBA has taught me quite a lot and I’m keen to find a new place to do some more learning.

If you have read my blog, visited my GitHub repositories, or seen some of my presentations you already know a bit about me. I’m a decent engineer with pretty good skills at Python and the shell and I get the job done. I’m a quick learner and a voracious reader of tech sites.

But let me quote from my CV.

“I design and build systems that allow organisations to deploy and secure Macintosh computers, iPhones and iPads.”

“My personal mission is to build a back end that supports well designed, well built, and fuss free devices for the end user. I’d like to work for an organisation that has the same goal and believes Apple devices are a great way of delivering that.”

My CV can be found at https://bit.ly/Tony-CV-2024. If you hear of anything drop me a note at honestpuck at gmail dot com

Another Solution for Uploading Scripts

Posted on by

Some time ago I wrote Scriptorium to keep the scripts on your Mac in sync with the same scripts on your Jamf Pro server.

Thanks to the excellent tool jctl from the people at the Marriot Library of the University of Utah I have a simple solution using their tool and git hooks so when you push to your repo the script is updated in Jamf Pro.

Install jctl from their repo at https://github.com/univ-of-utah-marriott-library-apple/jctl/wiki/Installing. Once you have done that you need the hook in .git/hooks in your repo. Call it pre_push

#!/bin/bash
if [[ $(git diff origin/main@{1} origin/main --name-status | grep -c ${name_of_script}) > 0 ]]; then
    var=$(cat "${path_of_script}")
    /opt/homebrew/bin/jctl scripts -i ${id_of_script} -u script_contents="${var}" --use-the-force-luke
fi

pre-push means that the hook runs after you have typed git push and before the actual push occurs. If you want to you can stop the push from happening by returning non-zero from your script. I don’t do that as I’m not doing any testing in the hook.

name_of_script is, of course, the name of the script. path_of_script is the name of, and path to, the script. id_of_script is the ID in Jamf Pro of the script, you can find it in the URL of the script.

I use direnv to set the variables. You could hard code them but that seems like actual work. Now every time you do a git push the script is sent to your server.

Let me know if you find this useful.

JNUC is Next Week

Posted on by
From https://publicdomainpictures.net/

JNUC 2023 in Austin is just around the corner. Come see my presentation on shell programming. https://bit.ly/jnuc2023.

From 5-line extension attributes to security shell scripts that run to over a thousand lines, shell programming is a key part of macOS administration. We can all write better code.

Through examples of shell code attendees will learn:

  • How to avoid bad code
  • How to make code comprehensible
  • How to write resilient code

Solve The New York Times Spelling Bee

Posted on by

I am currently visiting my brother in Maryland and his wife, Michele, likes doing the NYT Spelling Bee. It’s a word puzzle where you get 7 letters and one of the 7 is required, with a minimum length of 4 letters. Letters can be repeated in a word. There is at least one word that uses every character.

This raises the question, given a list of words what is the shortest piece of code to give all possible answer words. I think I’m close with a single awk command.

bee() { ;  awk  "/^[$1]{4,}$/ && /$2/" /usr/share/dict/words ; }

You call the function bee <letters> <letter required>. For example the puzzle on September 12 is bee gviotnp t.

As I cannot get the list of words the NYT is using I am using the word list on my Mac which returns more words than the NYT. If I could find a shorter word list closer to the NYT list it would improve the output.

Looking for a new role

Posted on by

Excuse me for a minute, I’m going to do some shameless self promotion.

Back in March last year I asked for your help in finding me a new role. That search did end up with me getting a job. Unfortunately it wasn’t the good fit I was looking for and I will be finishing up there in four weeks.

I’m currently hard at work looking for my next role. I think my time at my current employer is coming to an end and to become a better engineer I need to move to somewhere new. I need a place that is a better fit.

If you have read my blog, visited my GitHub repositories, or seen some of my presentations you already know a bit about me. I’m a decent engineer with pretty good skills at Python and the shell and I get the job done. I’m a quick learner and a voracious reader of tech sites.

But let me quote from my CV.

“I design and build systems that allow organisations to deploy and secure Macintosh computers, iPhones and iPads.”

“My personal mission is to build a back end that supports well designed, well built, and fuss free devices for the end user. I’d like to work for an organisation that has the same goal and believes Apple devices are a great way of delivering that.”

If you know of an organisation looking for a Client Platform Engineer or Device Management Engineer then I’d love you to drop me a note at [email protected]. If you want my CV to pass along you can grab it on my Github repo at https://bit.ly/tonyw2023.

Thanks for letting me advertise to you (again).

Using the NIST macOS_security project with Jamf Pro

Posted on by

I have just completed a project to update the security posture for a Mac fleet and wrote up a HOWTO use the NIST project with Jamf Pro. It is an impressive tool and well maintained by a number of people.

My HOWTO details policies, profiles, extension attributes and scripts. I hope people find it useful. Check it out at https://github.com/Honestpuck/NIST-macos-security-HOWTO

Finding A Good Fit

Posted on by

Excuse me for a minute, I’m going to do some shameless self promotion.

I’m currently hard at work looking for my next role. I think my time at my current employer is coming to an end and to become a better engineer I need to move to somewhere new. I need a place that is a good fit

If you have read my blog, visited my GitHub repositories, or seen some of my presentations you already know a bit about me. I’m a decent engineer with pretty good skills at Python and the shell and I get the job done. I’m a quick learner and a voracious reader of tech sites.

But let me quote from my CV.

“I design and build systems that allow organisations to deploy and secure Macintosh computers, iPhones and iPads.”

“My personal mission is to build a back end that supports well designed, well built, and fuss free devices for the end user. I’d like to work for an organisation that has the same goal and believes Apple devices are a great way of delivering that.”

If you know of an organisation looking for a Client Platform Engineer or Device Management Engineer then I’d love you to drop me a note at [email protected]. If you want my CV to pass along you can grab it at https://www.dropbox.com/s/tym5d0kzk6fzi3q/TonyWilliamsCV-2022.pdf?dl=0

Thanks for letting me advertise to you.

Designing Scriptorium

Posted on by

I just released Scriptorium, a small console program. Here are some notes on how I used argparse to do that.

We need a function to parse our arguments. Parsing is taking the line of words from the command line and processing them to extract the structure and meaning. The term ‘word’ can be fraught with complexity on the shell command line but a simple definition is any set of characters delimited by a space or matching quotes.

Scriptorium has a simple structure, scriptorium <command> [<argument>]. Some commands don’t have any arguments, for some arguments are entirely optional and for others arguments are required. But let’s start building that parser.

    """ build our command line parser """
    parser = argparse.ArgumentParser(
        epilog="for command help: `scriptorium <command> -h`"
    )
    subparsers = parser.add_subparsers(description="", required=True)

You can also see we are starting to build some help in ‘epilog’ – this is the final line printed when you run scriptorium --help. Then we need to have some code to parse each individual commands arguments. This is a subparser.

Continue reading →