Using AWK and R to parse 25tb
Recently I was tasked with parsing 25tb of raw genotype data. This is the story of how I brought the query time and cost down from 8 minutes and $20 to a tenth of a second and less than a penny, plus the lessons learned along the way.
Understanding propensity score weighting
Come enjoy a graphical exploration of various propensity score weighting schemes.
One year to dissertate
I've compiled some resources that I used when completing my dissertation and I wanted to share them with YOU! Throughout this post, I link to a bunch of different templates that I used throughout my process. You can find them all in a GitHub repo. This how-to has gotten a biiiiiit long. This post contains the whole kit-and-kaboodle, but I will also be releasing these in a series of smaller posts over the next couple of weeks.
p-value thoughts: A twitter follow up
A conversation about how "convincing" various studies were based on sample size and p-values led me to post a poll on twitter. Here I discuss some thoughts that came up based on these results. tl;dr: p-values are hard, twitter is a fun way to spur stats convos!
Shinyviewr: camera input for shiny
A brief intro to, and tutorial for, the new function in the shinysense packages: shinyviewr. This function allows you to take photos using the camera on your computer or phone and directly send them into your shiny applications.
Bringing the family together: Finding the center of geographic points in R
My husband's family throws a family reunion every year and this year we've been tasked with co-planning it. We were trying to decide on the best location for everyone, so I embarked on a mission to find the center of all of our residences.
R release names (Updated)
I always love discussions about R release names and their origin. I have been working on this list for a while -- with the release of "Short Summer" today, I thought it'd be a good time to post!
network3d - a 3D network visualization and layout library
Recently, I have found myself needing to visualize networks. There are plenty of lovely options in R for visualizing networks in 2d, but I have found that many of the networks I want to visualize work much better when done in 3d and here the options are much smaller. This has prompted me to build the package network3d. This post will be a brief intro to using it.
The United States of Seasons
How different is the warmest day from the coldest day all around the country? Using readings from 7,000+ NOAA weather stations across the country we can find out.
Wrangling Data Day Texas Slides
Since twitter threads are excessively cumbersome to navigate, Maëlle asked me to relocate the list of #rstats Data Day Texas slides to a blog post, so here we are!
A set.seed() + ggplot2 adventure
Recently I tweeted a small piece of advice re: when to set a seed in your script. Jenny pointed out that this may be blog post-worthy, so here we are!
A year as told by fitbit
Of all of the important things that happened in 2017, probably the most impactful on the world is that I managed to wear a fitbit the entire year. Here I download my entire years worth of heart rate and step data to see what my 2017 looked like, in terms of heart beats and steps.
Leveraging uncertainty information from deep neural networks for disease detection - a summary
I was recently sent this fantastic paper on using uncertainty in deep neural networks. In it the authors demonstrate a practical use of approximate bayesian inference by dropout in the context of massively complicated computer vision models for diagnosing disease. The paper, while well written, is very long. Here I summarize it into its main points and comment on their impactfulness.
Secret Sampling 🎅🤶
'Tis the season for white elephant / גמד וענק / Yankee swap / secret santa-ing! We thought it'd be particularly fun to do it #rstats style.
Thanksgiving Gantt Chart
Thanksgiving 🦃 is right around the corner 🎉 -- this year we are hosting 17 people 😱. If you too are hosting way more than your kitchen normally cooks for, perhaps this will be of interest!
LSTM neural nets as told by baseball
One thing I always found confusing when learning what an LSTM does is understanding intuitively why it's doing what it does. Here I attempt to give an example of how a LSTM hidden layer can be thought of through baseball.
MCMC and the case of the spilled seeds
For a long time I was confused by MCMC. I didn't understand what it was, how it worked, and why we needed to do it. In this post I attempt to clear up those questions and allow you to play with the Metroplis Haystings algorithm as it attempts to find a posterior to help solve a mystery of two messy birds.
R release names
I always love discussions about R release names and their origin. I have been working on this list for a while -- with the release of "Short Summer" today, I thought it'd be a good time to post!
Commentary and follow up to p<0.005 suggestion
A recent paper, Redefine Statistical Significance by 72 co-authors, has caused quite a stir in the statistical community. Our student-run journal club at Vanderbilt will be discussing this contribution at our meeting led by Nathan James this week, so I've attempted to create a list of significant responses/commentary that have come out since this paper was posted on PsyArXiv.
The traveling metallurgist
Here I attempt to explain the concepts behind the optimization technique simulated annealing and the combinatorial optimization problem of the traveling salesman. First in words, and then more excitingly in an interactive visualization.
A Simple Slack Bot With Plumber
I've been excited about the R package Plumber ever since hearing about it for the first time as useR2017. So when I finally found an application that would allow me to use it, sending cat and dog photos over slack, I jumped at the opportunity.
The Exponential Power Series
I find series expansions fascinating. I also find any math envolving e to be fascinating. Here I explain some of the facets of the exponential power series and its connection to my favorite distribution, the Poisson.
Why you maybe shouldn't care about that p-value
Recently, there seems to have been an uptick in citations of studies or statistics about this or that in the news and on the internet. Often these studies claim validity on the basis of a p-value. Through a small contrived example I make the point that in some situations we may want to ignore the forest and focus on the trees.
How to make an R Markdown website (with RStudio!)
Interested in creating your personal website with R Markdown? We've updated our R Markdown website tutorial to depend on RStudio for simplicity, making website building easy as 🍰!
New and improved draw charts in shinysense
Recently I overhauled the drawr function of my package shinysense. Some bugs were fixed but potentially more interesting new features were added. Among these are support for time series and the ability to use the function outside of Shiny. This post covers what changed and how to use the new features.
Twitter trees
A little over a week ago, Hilary Parker tweeted out a poll about sending calendar invites that generated quite the repartee. It was quite popular -- so much so that I couldn't possible keep up with all of the replies! I personally am quite dependent on my calender, but I was intrigured to see what others had to say. This inspired me to try out some swanky R packages for visualizing trees.
The making of "We R-Ladies"
Maëlle and I created a mosaic of R-Ladies for the JSM Data Art Show. Here is a quick tutorial if you are interested in trying something similar!
Happy World Emoji Day: an analysis of rOpenSci's Slack emojis
HAPPY world emoji day! In honor of this momentous occasion, I have decided to analyze the emojis used on rOpenSci's Slack.
Introducing the tuftesque blogdown theme
If you like the way our blog looks, you too can have your own blogdown driven site just like it! In this post I walk through how to set up an RMarkdown driven blog from scratch using blogdown and the tuftesque theme constructed for Live Free Or Dichotomize.
useR!2017 digressions
We both recently attended useR!2017 in Brussels. It was a blast to say the least. Here we will cover our favorite things about things about the conference and the lessons we learned.
runconf17, an analysis of emoji use
I had such a delightful time at rOpenSci's unconference. Not only was it extremely productive (21 packages were produced!), but in between the crazy productivity was some epic community building.
ENAR in words
I had an absolutely delightful time at ENAR this year. Lots of talk about the intersection between data science & statistics, diversity, and great advancements in statistical methods. Since there was quite a bit of twitter action, I thought I'd do a quick tutorial in scraping twitter data in R.
Introducing shinyswipr: swipe your way to a great Shiny UI
Recently we have been working on a shiny app that mimics tinder for preprints. One of the more exciting things we've done in this app is implimented a swiping input. Now you can to with the package shinyswipr.
Intro to GMD
Lucy and I have made a simple package that allows you to pull down a collaborative google doc directly into an RMD file on your computer. Hopefully speeding up the process of writing collaborative statistical documents.
The dire consequences of tests for linearity
This is a tale of the dire (type 1 error) consequences that occur when you test for linearity 😱
The prevalence of drunk podcasts
For today's rendition of I am curious about everything, in Hilary Parker & Roger Peng's Not So Standard Deviations Episode 32, Roger suggested the prevalence of drunk podcasting has dramatically increased - so I thought I'd dig into it.
Yoga for modeling
A New Year's resolution for all of our models: get more flexible! By flexible, we mean let's be more intential about fitting nonlinear parametric models.
CatterPlot thank you note
Lara Harmon has put in countless hours to build and uplift the ASA Student community. We are SO grateful.
Custom JavaScript visualizations in RMarkdown
Recently RStudio added JavaScript chunks to RMarkdown. This makes many exciting things possible. Among these things is making your own custom JavaScript visualizations of data managed in R, all without leaving the .Rmd document. This is a quick walkthrough of doing just that.
Regression modeling strategies: a student's perspective
Nick and I are starting a series following Frank Harrell's Regression Modeling Strategies course. Get ready for some crazy fun.
dplyr thank you note
It's that post-holiday time of year to write some thank yous! I'm getting excited to attend rstudio::conf next week, so in that spirit, I have put together a little thank you using dplyr
Wait, what are P-values?
P-Values are annoying, let's understand them so we dont get beaten by them.
Hill for the data scientist: an xkcd story
This was inspired by Hilary Parker & Roger Peng's Not So Standard Deviations Episode 28. It was suggested that it would be useful to lay out Hill's criterion for data scientists, I agree!