Scientific Python at Microscopy & MicroAnalysis 2019
Today, I presented a talk titled “Scientific Python: A Mature Computational Ecosystem for Microscopy” [PDF] at the Microscopy and MicroAnalysis conference in Portland. A few members of the audience familiar with scientific Python told me they had learned something, so I’ll highlight the few topics that I think may have qualified. SciPy 1.0 paper The first official release of SciPy was in 2001, and a mere 16 years later we reached 1.
Designing for Intelligent Augmentation
The first issue of the Harvard Data Science Review was published last month. I enjoyed Michael Jordan’s opinion piece, in which he considers progress made (and, yes, hype!) in so-called artificial intelligence. Particularly, I was excited about the phrase “Intelligent Augmentation”—the pursuit of systems that enhances our ability to reason about a subject, by providing additional analysis, data, and emphasis. Jordan writes: IA will also remain quite essential, because for the foreseeable future, computers will not be able to match humans in their ability to reason abstractly about real-world situations.
Linking to emails in org-mode (using neomutt)
Update 2018-11-2: Change the URL scheme to message://. See “Other Systems” below. org-mode is, to me, is one of the most valuable parts of the emacs ecosystem. I use it to take notes, plan projects, manage tasks, and write & publish documents. Nowadays, a lot of work arrives via email, and so it is helpful to be able to refer to messages directly from my notes or lists of tasks.
Search headers in org-mode
In org-mode, I often have the need to jump to a top-level heading matching some word. Since an org-mode buffer can be searched just like any other, I can simply invoke forward search with C-s, but this will match all occurrences of the text, instead of limiting the search to headings only. This makes it hard to search for a phrase like “Travel”, for which I have a top-level heading, but also often occurs elsewhere in my notes.
WebSockets in Python (and some Redux)
As part of our work on Cesium and its web frontend, we’ve developed an easy mechanism for Python web developers to push messages from their Python backends to the browser. There are plenty of potential use cases, but consider, e.g., that you want to verify a credit card number submitted by your user. Traditionally, you’d submit the number, and then poll the backend repeatedly from the browser. Not very elegant :/
Compile TensorFlow from source with gcc6.1
Due to two bugs in gcc 6.1 affecting the re2 library (one of which has been fixed in 6.2)), TensorFlow cannot be compiled from source one some systems (including my Debian Testing install). To work around the issue, modify tensorflow/workspace.bzl and change the re2 description to: native.git_repository( name = "com_googlesource_code_re2", remote = "https://github.com/stefanv/re2.git", commit = "86503cb89d82b723ae0bce35e1e09524910cd319", ) The re2 library is now downloaded from my fork, which applies a one line patch.
Amazon owns your ebooks
Python & Matplotlib on OSX
One day, we will hopefully have a grand unified build and package management system for Python where everything is free & open and Just Works ™. Until then, you have two options: brew Python + pip brew install python3 pyvenv -v ~/envs/py3 source ~/envs/py3/bin/activate pip install matplotlib Pros/cons: Pip is the standard Python package management tool, and uses the official Python Package Index (PyPi) repository. Wheels on PyPi are built by authors themselves from open recipes.
VPN-only access to the internet
A few days ago, I noticed that my outbound email wasn’t being delivered—it seems as if my ISP blocks access to some outbound ports. An easy workaround is to route traffic through the work VPN, but knowing myself I’d forget to switch it on, leaving my outbound mail stranded. I needed a way of blocking any access to the internet, unless it was leaving through the VPN (since I was sure I’d notice that pretty quickly).
Visualizing and pruning the docker image tree
Visualizing the Docker image dependency tree is a useful way of checking image sizes and inter-image dependencies. Since images occupy a lot of disk space, we may also want to prune unused images. Visualizing image dependencies In version 1.2.0 of Docker, the image dependency tree is available via the docker images --tree command: $ docker images --tree Warning: '--tree' is deprecated, it will be removed soon. See usage. ├─511136ea3c5a Virtual Size: 0 B │ ├─5bc37dc2dfba Virtual Size: 192.