Endjin - Home

Architecture

Long Running Functions in Azure Data Factory

by Jess Panni

Azure Functions are powerful and convenient extension points for your Azure Data Factory pipelines. Put your custom processing logic behind an HTTP triggered Azure Function and you are good to go. Unfortunately many people read the Azure documentation and assume they can merrily run a Function for up to 10 minutes on a consumption plan […]


So, this week I’m moving on from the creational patterns (though I might return at some point) and moving on to the structural design patterns! Structural design patterns are ones that deal with the relationships between different elements. Essentially, they define how different parts of the code interact with and relate to each other. This […]


Here at endjin we spend a lot of time working with data, and securing that data is top on our list of priorities . Therefore, anything we can do to reduce the need for storing access keys is a huge win! (Here is a guest blog from Barry Smart at Hymans Robertson which details our […]


I’ve been focusing lately on code quality, refactoring and architecture. A large part of my reading (& Pluralsight course-ing) has been based around design patterns, and how the application of these patterns can help bring your code in line with the SOLID principles. This means that code which uses these patterns is often more extensible, […]


Firstly, I should start by calling out the fact that there are different types of architecture. For example, enterprise architecture focuses on strategy rather than code. It is the strategy for how people, processes, technology etc. is utilised across an organisation in order to be effective and efficient. The type of architecture I will be […]


Agile Architecture: Painting by Numbers

by Howard van Rooijen

I originally wrote this post in 2007 (now lost in the annals of time), and is one of my favourite posts. I’m very fond of the confluence of ideas and the serendipity of discovering someone amazing from another discipline articulating the same thoughts I was having. During one of the Open Space sessions I attended […]


In the last post I explained how to create a set of Azure Functions that could load data into Snowflake as well as execute Snowflake queries and export the results into your favorite cloud storage solution. In this post I will show how we can use these functions in Azure Data Factory to plug Snowflake […]


If, like me, you are a fan of Azure Data Factory and love Snowflake then you are probably disappointed that there isn’t a native Data Factory connector for Snowflake. While we wait for an official connector from Microsoft we have no alternative but to roll our own. In this blog post I will walk you through […]


For those who don’t know, I am currently in the process of carrying out a full code review and improvement of some of our internal code using NDepend. To find out more about the quality measures that NDepend uses to analyse the code – read my first blog in this series! Otherwise, let’s press on! […]


Over the years we’ve built a lot of IP up here at endjin. With a growing codebase, and with my training shifting to look at software architecture, I thought I’d attempt a code review to see what insight I could gain! So… Here goes nothing! The tool I chose to carry out this review is […]


And here we are, the culmination of a 3-month long journey deep into the realms of Rx operators. If any of you have been here since the beginning, I hope you have enjoyed going on this ride with me, and if any of you are just starting now, here’s my weekly shameless plug for the […]


We’ve had ongoing issues when deploying web and functions apps involving the locking of DLLs during the deployment. The specific case I’m going to talk about focuses on Azure Functions, but you can also run Web Apps from a package (though the Azure Pipelines tooling currently only works for functions, so you would need to […]


So, this week we are looking at the Buffer and Window Rx operators. (If you have no idea what I’m on about, I suggest you start at the beginning!) There are a few different implementations of these operators, and we are going to focus on the time-based versions. In order to do this, we need […]


In September I joined endjin a Technical Fellow (an entirely new branch in endjin’s career pathway to accommodate me – more on that later). I’ve has been involved with endjin since 2011, as an Associate, helping to deliver some of our most technically challenging projects (and if you go even further back, I attended Cambridge University with endjin co-founder […]


In case you missed it… Here’s a link to my last blog on understanding Rx (luckily this blog has an internal buffer so if you’re just tuning in now, you’ve not missed your chance)! OnNext(Understanding of the Rx operators) Now one of the most exciting things about Rx is that it has its own implementation […]


Overflowing with dataflow part 2: TPL Dataflow

by Carmel Eve

Edit: In case you missed it! Here’s a link to part 1, a general overview of dataflow as a processing technique! The specific implementation of dataflow that I want to talk about is the TPL dataflow library. The task parallel library is a .NET library which aims to make parallel processing and concurrency simpler to […]


Overflowing with dataflow part 1: An overview

by Carmel Eve

In a recent project, I was asked to produce a tool for importing a fairly large amount of data at once, this data then needed to be processed and exported. After much refactoring, I achieved a solution I was satisfied with which used TPL dataflow to execute the processing in parallel. Before I talk specifically […]


(TL;DR This month I learnt a lot about the importance of package management, maintenance and the minimisation of dependencies.) It’s been a little while since my last blog, I’ve been pretty bogged down in a world of packages, .NET frameworks and testing. And it all started with what I thought would be a simple update […]


I’m very excited that Ian Griffiths has joined endjin as a “Technical Fellow”. This is a new career pathway branch we created especially for Ian, as he didn’t really fit into any of our existing roles; his skills and expertise exemplify a pathway that many software engineers desire, but few have the opportunity to achieve […]


Digital Transformation is the buzzword of the moment; organisations are trying to reconfigure themselves to operate in the digital age. One of the key enablement strategies involves putting APIs at the heart of the organisation to centralise access to data, logic and value, while removing those traditional barriers that prevent collaboration between organisational silos. APIs […]