A scalable solution for Datawrapper vizualisations with Data Factory and Databricks

During the Covid pandemic we at NHO (The Confederation of Norwegian Enterprise) have performed surveys with our members, which are Norwegian enterprises. We have asked them how business is going, if they fear bankruptcy, how the income is, and how they look at the next months going forward. Media and others have wanted this data […]

Setting up Azure DevOps for your Data Factory – Part 3/3 – Create release pipeline

Now we have set up our environments and we have developed something that we want to release to test. This final post describes how to do that, and a few useful tips as well. Release processThe release process will have these steps;1. Stop any active triggers. We do not want any pipelines to start as […]

Setting up Azure DevOps for your Data Factory – Part 1/3 – Create Environemnts

We recently set up Azure Devops for deploying Data Factory, and I though I’d share some practical tips for how to do this. There are some great other resources out there for this, but I’ll try to put it all together for you here, and add a few bits I have found myself. This post […]

Comparing Databricks, Mapping Data Flows and SSIS as data transformation in ADF

I presented a session on SQLSaturday Oslo last weekend and can share some of the key takeaways here. This is by no means a complete comparison of every feature of these techniques, but show some of the strengths and weaknesses and can be a start when considering what tech to use in your Azure Data […]

Starting and stopping SSIS Runtime with Azure Functions in ADF

SQL Saturday Oslo is approaching and I’m giving a session where I compare SSIS, Databricks and Mapping Data Flows for data transformation in Data Factory. One of the problems I got when setting this up is that when running SSIS in Data Factory you must handle the Integration Runtime and start (and stop) this efficiently. […]

Using Databricks delta to speed up Azure SQL load

If your organization doesn’t have enough data to require Azure SQL Warehouse with Polybase loading from data lakes, you might have observed that loading much data with Azure SQL databases can take some time. Today I show an example of how to use Databricks delta together with stored procedures to speed this up. The case […]