Endjin - Home

Architecture

Have you or are you about to invest in Azure Databricks? If so, the new Spark offering in Azure Synapse Analytics is likely to have grabbed your attention and rightly so. Why is Microsoft putting yet another Spark offering on the table and what does it mean for you?


For years we have been building modern cloud data solutions on Azure and helping our customers transform their use of data to drive outcomes. Here are 5 reasons why Azure Synapse Analytics might just be the service that we have been crying out for.


Jess and Carmel recently gave a talk at Azure Oxford on “Combatting illegal fishing with Machine Learning and Azure – for less than £10 / month). The recording of that talk is now available for viewing!

The talk focuses on the recent work we completed with OceanMind. They run through how to construct a cloud-first architecture based on serverless and data analytics technologies and explore the important principles and challenges in designing this kind of solution. Finally, we see how the architecture we designed through this process not only provides all the benefits of the cloud (reliability, scalability, security), but because of the pay-as-you-go compute model, has a compute cost that we could barely believe!


Wardley Maps are a fantastic tool to help provide situational awareness, in order to help you make better decisions. We use Wardley Maps to help our customers think about the various benefits and trade-offs that can be made when migrating to the Cloud. In this blog post, Jess Panni demonstrates how we used Wardley Maps to plan the migration of OceanMind to Microsoft Azure, and how the maps highlighted where the core value of their platform was, and how PaaS and Serverless services offered the most value for money for the organisation.


The application of scientific experimental process to software development leads to the development of fully-validated solutions. This approach provides you with confidence in designs and means that you can quickly identify ideas which are not worth pursuing.

At endjin we use the ideas of hypotheses and experimentation when designing any solution and this gives us full confidence in the designs we produce. In this post we outline the steps and advantages of using this approach.


An Overview of the Azure CNAB Quickstarts Library

by Howard van Rooijen

The Azure CNAB Quickstarts Library helps you get up and running with CNAB and Porter. We’ve built quickstarts covering solutions like WordPress, Ghost, Mattermost, and data platforms like Apache Airflow, SQL Server AlwaysOn clusters and Kubernetes features like an nginx ingress controller and an Azure AD enabled OAuth2 Proxy. We’ve condensed all our learnings from the past 9 months of working on the project and turned them into a 10 minute video which explains all the key concepts. We hope this video helps accelerate your own CNAB & Porter epiphanies!


In partnership with Microsoft, we have released Azure CNAB Quickstarts Library on GitHub. CNAB (Cloud-Native Application Bundle) is a new specification designed for facilitating the packaging, installation, upgrading and uninstallation of cloud-native solutions in the cloud, on-premise or on the edge. We’ve created a number of quickstarts covering Apache Airflow, Azure Kubernetes Service, Ghost, Kubeflow, SQL Server Always On and Wordpress to help demonstrate the power of CNAB and Porter.


In this post we show how a combination of Kubernetes, Azure Durable Functions and Azure API Management can be used to make legacy batch processing code available as a RESTful API. This is a great example of how serverless technologies can be used to expose legacy software to the public internet in a controlled way, allowing you to reap some of the benefits of a cloud first approach without fully rewriting and migrating existing software.


NDC London Day 1

by Ian Griffiths

In this post, Ian describes some of the highlights from the NDC London conference


With a variety of integration support through client SDKs, PowerShell cmdlets and REST APIs, it can be hard to know where to start with integrating Azure Analysis Services into your custom applications. This posts walks through the options, and lays out a simple guide to choosing the right framework.


In this blog from the Azure Advent Calendar 2019 we discuss building a secure data solution using Azure Data Lake. Data Lake has many features which enable fine grained security and data separation. It is also built on Azure Storage which enables us to take advantage of all of those features and means that ADLS is still a cost effective storage option!

This post runs through some of the great features of ADLS and runs through an example of how we build our solutions using this technology!


Survey: Expression Trees use cases in .NET

by Howard van Rooijen

Expression Trees are one of .NET’s super powers; but they are hugely overlooked and misunderstood. They can be used for on-the-fly code generation; building DSLs, and producing high performance computational algorithms. I’m trying to gather evidence about how you are using them, to make a case for future investments in the Expression Tree subsystem in .NET Core / .NET 5.0


Long Running Functions in Azure Data Factory

by Jess Panni

While on first inspection Azure Function look like a good candidate for long running operations as they can run for 10 minutes on a Consumption plan, HTTP Triggers only run for 230 seconds because of a limitation with the Azure Load Balancer. In this blog post Jess Panni demonstrates how Durable Functions can be used instead.


This blog is part of a series around design patterns. This post focuses on the composite pattern. The composite pattern is often used in situations where you want to me able to treat groups and individuals in the same way during processing.


Building a secure solution on Azure can be a daunting task. Using Azure Functions and Managed Identities, we have built up a pattern for giving services access to one another, woithout the need to store credentials. These managed identities can be given access to necessary resources. For example, they can be granted roles and added to access control lists in ADLS Gen2 accounts, or the ability to access keys in key vault. This means that data can be securely accessed without needing to store connection strings or app passwords.


This is the first blog in a series about design patterns. This blog focuses on the differences between the factory method and abstract factory patterns. The factory method is a method which takes the creation of objects and moves it out of the main body of the code. An abstract factory is similar to the factory method, but instead of a method it is an object in its own right.


Software architecture and Agile project planning are often seen to be at odds. However, here at endjin we think that the way in which they intersect solves a lot of the common issues surrounding architecture. The key to successful architecture is constantly keeping the drivers in mind and having a tight communication loop as the architecture is implemented. These concepts are ones which are key to agile project management, and the combination of these two disciplines can be extremely powerful and successful when applied correctly.


Agile Architecture: Painting by Numbers

by Howard van Rooijen

I originally wrote this post in 2007 (now lost in the annals of time), and is one of my favourite posts. I’m very fond of the confluence of ideas and the serendipity of discovering someone amazing from another discipline articulating the same thoughts I was having. During one of the Open Space sessions I attended […]


In the last post I explained how to create a set of Azure Functions that could load data into Snowflake as well as execute Snowflake queries and export the results into your favorite cloud storage solution. In this post I will show how we can use these functions in Azure Data Factory to plug Snowflake […]


If, like me, you are a fan of Azure Data Factory and love Snowflake then you are probably disappointed that there isn’t a native Data Factory connector for Snowflake. While we wait for an official connector from Microsoft we have no alternative but to roll our own. In this blog post I will walk you through […]


1 2 3