Writings

Run system commands from your scripts

Just a quick tip: You can probably run shell commands from your favorite programming language.

PSA: GAPIT Containerization for HPC

I ran into a thorny problem running GWAS on an HPC in December so I’m sending out this PSA to hopefully prevent someone else running into the same issue.

Write more small tools

You should write more small tools. At first this will be slower than doing the tasks manually but it’s wort the time so you can:

Sharing n-dimensional data between Python and R

Storing multidimensional data is a pain. In R or python you can save the n-dimensional array to disk, but not in a way that is easily accessible by the other. Admittedly…

Allow named arguments in R’s CLI

When run on the command line, a R script can be set to accept arguments by adding to the script

SQL Strikes Back

For additional context please see Part 1 Part 2

Running GWAS en masse.

As part of a project I’m about to run a lot of analyses. Below are some initial tricks I’ve figured out with (hopefully) more posts like this to come as I find more…

Tip: Make your life easier with Symbolic Links in WSL

Modern Windows machines can give you access to Linux command line tools via the Windows Subsystem for Linux. On launch the present working directory is set to the…

Building a GWAS Container

Today we’re going to look at building a container to run a GWAS¹ on a computing cluster. Container construction can be rigorous and necessitate a fair bit of understanding…

Quick Tip: Write notebooks, run scripts

Much of what I write is in notebooks (thanks to the lovely nbdev library) but there are times where this is not convenient. For instance, while tuning hyperparameters or…

Tip: Use `R` to create an email nudge

Suppose you’re giving a presentation and you want to make it easy for people to contact you afterwards. Maybe you have your email in the acknowledgements or maybe you make…

Go Read About R’s Function Call Semantics

This is the sort of thing you don’t realize until it would be really useful to access the name of a variable or run text as if it were code. I think the most accessible…

Worse is better case study 1

Slurm¹ is a tool commonly available on computing clusters for scheduling job. For a while I’ve wanted a local instalition of it so I can queue computational experiments to…

Worse is better and not doing things “right”

tacit knowledge

“Worse is better” is an idea I get a lot of mileage out of. Here’s the crux of it:

Use a spreadsheet to manage your CV & Resume

professional development

For the past year I’ve been using a spreadsheet and RStudio to manage my resume and curriculum vitae. This post is a pitch for why you might want to do this and an overview…

Simulation as a Super Power

Writing simulations is one of the best ways I’m aware of to build one’s statistical intuition and comfort with data visualization. In addition to being able to try out new…

Solving the Wrong Problem

tacit knowledge

There’s a perspective that “it is better to know nothing than to know what ain’t so.”1 In my experience this is certainly the case with debugging because “knowing” will lead…

Making a “Visible” Neural Network

In most neural networks, neurons are not parametric in the same way that linear models are. In a image recognition model there may be neuron which functions to detects edges…

Save only what you need

tacit knowledge

Collecting data from many sites is expensive but using a biophysical model, many sites and years can be ‘harvested’ in minutes. I’m building a dataset with many cultivars…

Capturing Tacit Knowledge Through Blogging

tacit knowledge

One of my goals with mentoring is to help others avoid places I got stuck (or get through them more easily) when I was learning. Part of this is writing guides…

Tips: Do-nothing Scripting in Bash

Do-nothing scripting is a nice way to blend documenting a protocol with running it. You can use this template as a place to start:

A Comment is a Comment.

R treats “#” as a comment in text files.

Trivia: R can have Comments in Tables

R allows for comments to exist in tables. If there’s a # in the table you’re reading (e.g. as a part of a column name like chromosome#) then it can cause an unequal number…

Tips: Open a new Interactive Session in Running Session

A handy trick with batched processes on an HPC is that you can start an interactive session in a running session. Here’s an example where I needed to check if I was nearing…

Trivia: In Python Missing Isn’t Equal to Itself

Python quirk I just learned and think is worth sharing. A missing valued doesn’t equal itself.

ggplot and the invisible hex code.

If you use an eight digit hex code for specifying a color value, the first two control transparency. Thus, you can set fill = “#00000000” in ggplot to get a boxplot with no…

Tips: Cache Intermediate Results with `pickle`

Here’s a useful pattern I’ve been getting a lot of mileage out of lately. If you’re running an analysis that has a time consuming step you can save the result as a python…

Tips: For those coming from R: Silent In Place Replacement

Silent, in place assignment updating an object This tripped me up even though it’s consistent with how I’ve seen other objects behave. I needed an attribute to hold data…

Approximate String Matching

fuzzywuzzy is a tool that isn’t necessary most of the time but when it is it can save a ton of time. It lets you do approximate string matching. I’ve used it for handling…

Tips: Reusing Custom Functions

Amendment: For packaging functions also see nbdev.

Tips: More Readable Data with `pretty-print`

Here’s a tool that some may find useful when working with data that’s not yet in a [DataFrame]. It lets one “pretty-print” an object making any text that would wrap easier…

Tips: Find the Graph you want in using a Graph Gallery

This or similar sites can be helpful for looking up the right code/library for a plot. You can also find library specific ones. (matplotlib, plotly)

Tips: Jupyter Plugins

I came across a handy set of tools for jupyter. There are of extensions for the notebooks that give you access to code snips, autocomplete by default, rendering a notebook…

Using Simulations to Check Your Statistical Intuition

R’s distribution simulation functions (e.g. dbinom, runif) make it quick and easy to double check one’s intuitions. For example, I’d been thinking that under H0 the…

ggplot Reordering a Discrete Axis

Reordering a discrete axis in ggplot after generation a lot simpler than one might expect. Rather than converting a character column to a factor (what if the data gets…

ggplot: Parsing Expressions

A useful trick is to pass expressions into ggplot. Here I’ve used the following as arguments in labs().

ggplot tips: markdown for italics in plots and adding breaks to color scaling

Two tricks today: 1. Scale fill functions can accept breaks and limit arguments so you don’t have to use hacky workarounds like binning the data before plotting (which is…

Workaround for Plotting Dendrograms

I think I have a solution to get decent enough dendrograms without fussing with base graphics.

Thought Experiment: Comparing Replicate Experiments’ Conclusions

A collaborator and I just talked through how one might compare two sets of p values. We’re thinking about the following approach.

ggplot frienndly corrlation plots with `ggcorrplot`

A co-worker ran into issues with corrplot::corrplot() cutting off the title. A useful alternative is ggcorrplot. It makes okay plots with ggplot2’s logic. Not as clean as…

ggplot font customization

You can set ggplot’s font using the theme function. Particularly if combined with functions from ggthemes or ggsci you can get very pleasing visualizations quickly. Beyond…

Installing Local Libraries Post Update

I just ran into the same issue after updating R on OSX — none of the libraries are associated with the new version. I tested and abandoned directly copying the libraries…

Diagrams as code

Check out DiagrammeR (or mermaid) if you need clean easy flow diagrams. I find they’re not to hard to make and even easier to forget that you’ve made them.

Roman numeral convenince function

For all you classics folks out there, R has a convenience function just for you! (As long as your numbers aren’t too big). Credit goes to Georgios Karamanis @geokaramanis for…

Resampling doesn’t need to be hard

As an example to show how accessible resampling can be, here’s bit of code that resamples an anova and computes an empirical p value.

Caching your enviroment and why you might not want to.

If you’re working on data that takes a long time to process, consider adding a the following to your analysis.

Data Type Matters In `gganimate`

Be mindful of your data types. Sometimes T == 1 == 1.0 (logical, int, double) but assuming these are equivalent can get you into trouble. For example, in these animations…

Document your functions with `roxygen2``

If you want to make future you more positively disposed to present you, then organize your work with R packages, save your custom functions in the /R/ directory. The function …

Reference current dataframe with `.`

In tidyverse you can use . to reference the current dataframe. This is really useful for plotting. For example, I’d like to plot channels In7 and In12 in that order but by…

Informal profiling with `tictoc`

The tictoc library has convenience functions for timing code. Here’s the basic usage relative to timing with base R.

`unlist` is handy and you should use it

Y’all should be using unlist(). unlist is a crazy handy function when you pair it with the purrr library. It’ll take a list and try to give you a vector. Strictly speaking…

ggplot `element_text()` colors

ggplot’s, element_text() comes with a color argument. Why does that matter? It doesn’t just accept atomics, you can hand it a vector! This effectively gives one access to…

Visualize the Influence of Outliers

If you’re looking at linear relationships, give this code a spin. It’ll automatically flag outliers based on 1.5*IQR and show you the fit with and without the outliers.…

Updating Results in Word from Table

Handy trick: Write your test results into a summary csv. Then while writing you can insert a link to the cell value for a result into your .doc. After that, changing post…

Greek Letters in ggplot

If you need to include greek letters or special characters in a plot use annotate() instead of geom_text(). On my machine it preforms a lot faster.

Shading with `geom_rect`

Use geom_rect() with the min/max set to -Inf and Inf to add a pleasant shading to your facets.

Don’t Ignore Your .gitignore

If you’re using git for version control, don’t forget about the .gitignore file. Anything large and static (like .abfs) or procedurally generated (e.g. plots) you can toss…

Cache Intermediate Results

Save R objects as rds. These could be works in progress or items that take a while to generate (e.g. re-sampled results). This gives the utility of saving a work space…

Tidyverse ’_at’ variants

tldr: tidyverse function variants take slightly different input. Testing out a variant or two can save you a lot of debugging time.

Syntax Tours: learnxinyminutes

Need a quick base R syntax lookup? Check out https://learnxinyminutes.com/docs/r/ . There are even examples with lm() and glm().

ggplot: Beware of Factors!

Don’t get suckered when converting factors! Numeric data can be assigned to a factor type which will throw a wrench in a plot or analysis (1/3)

Upgrading R versions on Windows

If you’re on windows, installr should allow you to copy over the libraries from previous versions.

Apply a theme to all your ggplots

If you’re applying the same theme to all your graphs, set it globally instead e.g. theme_set(ggplot2::theme_minimal()). If you have a lot of custom changes to your theme…

Clearing all but certain objects

If you run into a situation where you’re using a ton of memory (e.g. manipulating transcriptomic data, resampling, or working with electrophysiology traces) use rm() to…

Parallel Processing for `purrr` with `furrr`

If you’re iteratively making plots, resampling, or doing another task that would have your reach for a for loop or lapply() use furrr::future_map() instead. furrr gives…

Loading R functions from source

You can keep your working Rmd easier to navigate and less buggy by 1) packaging code into functions and 2) adding them to a companion Rfile. Load your functions with source()…