Azure Databricks Automated Testing - DZone Big Data

• Read original article here

We all know how important data testing is in this digital transformation world. ETL testing mainly consists of ensuring that data has safely traveled from its source to its destination. Data processing is prone to errors, and you may end up with some data loss, corrupted, or irrelevant data as a result of various issues during the transformation phase. This is why ETL testing is so important: it ensures that nothing has been lost or corrupted along the way.

To validate the data, the tester usually writes the ETL script or SQL by hand. The scripts will be run against the source and destination, and the results will be compared to validate the data. In this article, we'll look at how we can use Great Expectations, Databricks, and C# code to automate data quality and completeness tests.

Great Expectations is a shared, open data quality standard that helps in data testing. Expectations are data assertions. In Great Expectations, they are the workhorse abstraction, covering all kinds of common data issues. Expectations are declarative, adaptable, and scalable. They offer a large vocabulary for data quality.

Azure Databricks is an Apache Spark-based analytics platform and one of the leading technologies for big data processing, developed jointly by Microsoft and Databricks.

For the purpose of this tutorial, we are treating generic-food_source.csv as the source data set and generic-food_destination.csv as the destination dataset.

After successful installation, we can see that the library was successfully installed.

For the purposes of this tutorial, we have already created two data files that will serve as the source and destination in our case. In a real-world scenario, these will be various sources (on-premise, AWS, GCP, etc.) and destinations.

These two files have to be uploaded to Azure Databricks' dbfs file storage (use File->Upload Data option in the notebook):

Now you can see all the available tests under great expectation (dataframe.(ctrl+space)).

This will validate the row count between source and destination. 

We now have our test notebook ready, and all we need to do is run each cell from top to bottom to get the result. When you execute the last cell, you will get the output shown below, which will clearly indicate whether our test passed or failed, as well as some other useful information.

That's it! You've just finished one of your data validations, which can be reused to produce the desired results every time.

If you look closely, you can see the value for , which represents whether or not our test was successful. If the count does not match, this function will return . The actual record count can be found in the and the actual value . Isn't it simple and effective?

There are many more expectations (assertions) in the Great Expectations library that you can try out for yourself.

Now that we have a test notebook created in Azure Databricks, we will execute it from the code level and retrieve the results from Databricks. We'll be using C#, NUnit, and the Databricks client.

Before starting, we must make sure:

Give a project name and location to save the project. 

Click the Next button to complete the process. 

Right-click on dependencies in the project solution.

Search for "databricks" and install the Azure Databricks client.

Once this is installed successfully, we are ready with all our dependencies and can start coding to execute tests.

Once you create an NUnit test project, you can see that Visual Studio has provided a test class with some sample test code. We just need to edit this test class file and add some NUnit test annotations to organize the code.

Copy and paste the code below, replacing the parameters with your data.

When you pass the correct parameters and build the solution, you will see the test listed in Visual Studio's Test Explorer.

And there you have it! You have automated your first Azure Databricks test from Visual Studio. You can add to it by writing a number of test case validations.

Keep in mind that: