A couple of weeks ago I had the opportunity to attend NDC in Oslo. It was an absolutely brilliant experience, and my head is still reeling a bit from everything I learnt! The focus of a lot of the talks was around neural networks and machine learning – something which we have explored quite a […]

Have you been trying to create a Databricks cluster using the CLI? Have you been getting infuriated by something seemingly so trivial? Well, join the club. Although, get ready to depart the club because I may have the solution you need. When creating a cluster using the CLI command databricks clusters create, you’re required to […]

So, this week I’m moving on from the creational patterns (though I might return at some point) and moving on to the structural design patterns! Structural design patterns are ones that deal with the relationships between different elements. Essentially, they define how different parts of the code interact with and relate to each other. This […]

Here at endjin we spend a lot of time working with data, and securing that data is top on our list of priorities . Therefore, anything we can do to reduce the need for storing access keys is a huge win! (Here is a guest blog from Barry Smart at Hymans Robertson which details our […]

So, the other week I took a look at the Factory Method and Abstract Factory design patterns. This week I’m continuing with the creational design patterns, this time focusing on the builder pattern! The builder pattern is used when there is complex set up involved in creating an object. Like the other creational patterns, it […]

Whilst some of the Azure Active Directory PowerShell for Graph module (AzureAD) functionality has been rolled into the new Azure PowerShell Az module (Az), it’s not currently (and might never be) a replacement for the full power of what you can achieve with AzureAD. So, there’s every chance you’ll find yourself needing to use both […]

I’ve just completed my first year working full time for endjin, after I joined as an Apprentice II last May. And without sounding too much like a cliché, time has truly flown. In the ins-and-outs of daily working life I haven’t really noticed the huge progress I’ve made in the last year, but as we […]

I’ve been focusing lately on code quality, refactoring and architecture. A large part of my reading (& Pluralsight course-ing) has been based around design patterns, and how the application of these patterns can help bring your code in line with the SOLID principles. This means that code which uses these patterns is often more extensible, […]

As part of a recent project, we needed to automate the creation of a Power BI workspace. This workspace was part of a bigger deployment that provisioned Azure resources (including a Power BI Embedded capacity), and was orchestrated via an Azure DevOps Pipeline. Whilst there’s a tonne of useful information out there, there was no […]

I recently wrote a blog on using ADF Mapping Data Flow for data manipulation. As part of the same project, we also ported some of an existing ETL Jupyter notebook, written using the Python Pandas library, into a Databricks Notebook. This notebook could then be run as an activity in a ADF pipeline, and combined […]

I’ve very pleased to announce that endjin has become a Snowflake partner. This fantastic “designed for the cloud” data platform redefines what a data warehouse can be in the age of cloud. With features such as data sharing, usage based billing, and availability on Microsoft Azure, it has won our hearts. Over the last three years, we’ve […]

Five editions? Already? How time flies. The Power BI Weekly newsletter is proving a great success – we’ve just published the fifth edition, hundreds of people have subscribed and we’ve received lots of kind feedback. I think it’s safe to say that Power BI has become omnipresent in recent times. We use it widely here at endjin […]

Firstly, I should start by calling out the fact that there are different types of architecture. For example, enterprise architecture focuses on strategy rather than code. It is the strategy for how people, processes, technology etc. is utilised across an organisation in order to be effective and efficient. The type of architecture I will be […]

I originally wrote this post in 2007 (now lost in the annals of time), and is one of my favourite posts. I’m very fond of the confluence of ideas and the serendipity of discovering someone amazing from another discipline articulating the same thoughts I was having. During one of the Open Space sessions I attended […]

As part of a recent project we did a lot of experimentation with the new Azure Data Factory feature: Mapping Data Flows. The tool is still in preview, and more functionality is sure to be in the pipeline, but I think it opens up a lot of really exciting possibilities for visualising and building up […]

In the last post I explained how to create a set of Azure Functions that could load data into Snowflake as well as execute Snowflake queries and export the results into your favorite cloud storage solution. In this post I will show how we can use these functions in Azure Data Factory to plug Snowflake […]

If, like me, you are a fan of Azure Data Factory and love Snowflake then you are probably disappointed that there isn’t a native Data Factory connector for Snowflake. While we wait for an official connector from Microsoft we have no alternative but to roll our own. In this blog post I will walk you through […]

We launched Azure Weekly, a free weekly newsletter, back in 2014. 200+ issues and many thousands of global subscribers later, it’s still going strong. Last month Ed Freeman pointed out that analytics section of Azure Weekly was mainly full of Power BI articles and that it was by far the largest category in the newsletter. […]

For those who don’t know, I am currently in the process of carrying out a full code review and improvement of some of our internal code using NDepend. To find out more about the quality measures that NDepend uses to analyse the code – read my first blog in this series! Otherwise, let’s press on! […]

We recently had a requirement from a client that all of their Azure resources must be tagged with a specific set of tags, which were ultimately to be used for cost accounting when the bill came rolling in. For simplicity of this blog post, let’s assume the client just required that all resources had to […]