A Walk in the Tidyverse

• Read original article here

If you worked with R to explore a dataset and build a report from this analysis, you have probably heard about...

If you worked with R to explore a dataset and build a report from this analysis, you have probably heard about thetidyverse. If you used R in your data science project, to fit a predictive model able to produce the most accurate prediction possible for new data, you have probably experimented withtidymodels. However, questions like “what is the tidyverse” or “how does the tidymodels framework fit in it” might still have no clear answer for you.

What is the tidyverse

The simplest way to describe the tidyverse is as a collection of R packages – sharing common ideas and norms – designed to perform data science tasks easily and quickly. But I don’t like this definition too much because it tends to be too simplistic. tidyverse is more than that: it’s a philosophy or a lifestyle embodied in a collection of packages.

The tidyverse project – dating back to 2016 – has the ambitious mission of facilitating the conversation about data between humans and computers. This means that its structure and grammar wish to be as consistent and readable as possible, providing users with a step-by-step learning path in which every step (or package) makes it easier to learn the next one.

Tidymodels, similarly to tidyverse, builds on the core R language, by applying tidyverse principles to statistical modeling and machine learning domains.

The main principles of the tidyverse philosophy are three:

A large portion of R users is composed of data analysts and data scientists more than software developers. That’s one of the main reasons behind this principle, which implies not only providing clear documentation and training but also that the software itself should be intuitive and self-explanatory.

To achieve this goal, it’s essential to guarantee a friendly user interface and descriptive naming for packages and functions. The tidyverse approach is to rely on the grammar of verbs and nouns. For example, to filter from a dataset of students only the ones whose name is “Jenny” you will use:

And, when cleaning your dataset from “Not Available” values, you will write:

Whenever possible, functions in tidyverse re-use existing data structures, rather than creating custom ones. In particular, thetibbleis the preferred data structure for many R packages in tidyverse. A tibble is a modern reimagining of a data frame, “lazier” – i.e. it doesn’t change variable names or types, and doesn’t do partial matching – and “surly” – i.e. it prevents common R errors, like dropping dimensions. Data in tibbles is tidy, such that variables are in columns and observations are in rows.

Some other packages work at a lower level, focusing on a single type of variable. For example,stringrfor strings,lubridatefor date/times, andforcatsfor factors.

3. Design for the Pipe and Functional Programming

This principle relies on the use of themagrittr pipeoperator (%>%), enabling to chain together a set of R functipons, executed in a logical sequence.

For example, if you wish to compute the mean study time (in terms of hours) and the mean grade for the students of a class who passed or failed a certain course, you can use the following syntax:

where the column “Pass” is a Boolean indicating if the student passed or failed the course.

When it comes to data visualization, usingggplot2, a similar paradigm is applied to obtain an elegant graph, through the+operator. In fact,ggplot2enables the combination of independent components of a graphic in a series of iterative steps.

For example, if you wish to visualize a simple bar chart of the students’ grades you can use the following code:

where theggplot()function initializes a graphic,geom_col()adds a layer of bars whose height corresponds to the variables that are specified by the mapping argument, and ggtitle() adds a title to the chart.

Also, the tidyverse approach embraces the functional programming nature of R. This includes for example the use of immutable objects and copy-on-modify semantics, together with tools that abstract over for loops (i.e.apply()ormap()families of functions). 

The tidyverse project has as its specific goal to provide R users with a uniform interface, based on packages and functions that work together naturally and to close the gap between humans and machines, when working with data and statistical models.

To better understand this in practice, you could visualize the tidyverse and the tidymodels frameworks in a wider context, like the MLOps cycle below:

where tidyverse enhances the process of understanding and preparing the data for further processing, while tidymodels is an important tool to train and evaluate machine learning models.

Last, but not least, tidyverse is also the community of data scientists and R developers that use this collection of packages.

If you enjoyed this overview of the tidyverse and you are eager to learn more about the founding principles and the philosophy behind it, I strongly recommend readingTidy Modeling with Rby Max Kuhn and Julia Silge.  

If you wish to know more about how to use tidymodels and tidyverse functions to perform exploratory data analysis and build machine learning models, and you wish to practice with hands-on exercises in a pre-configured environment, you should check theCreate machine learning models with R and tidymodelslearning path onMS Learn.

Finally, if you want to get some guidance in your learning journey with R and Tidyverse by subject matter experts you can watch (live or on-demand) theTidyFridays Learn Live series.

Carlotta Castelluccio is a Cloud Advocate at Microsoft, focused on AI and ML technologies and passionate about their use in education. As a member of the Developer Relationships Academic team, she works on skilling and engaging educational communities to create and grow with Azure Cloud, by contributing to technical learning content and supporting students and educators in their learning journey with Microsoft technologies. Before joining the Cloud Advocacy team, she have been working as an Azure and AI consultant in the Microsoft Industry Solutions team, mainly involved in customer-face engagements focused on Conversational AI solutions.