Cloud & Engineering

We regularly write about our technical experiences (good and bad) and what we're learning from the market.

  • There are no suggestions because the search field is empty.

Tags

Write and deploy an Apache Beam pipeline with Dataflow

Posted by Sheng Wu on 02 April 2019

tech, gcp, dataflow, Apache Beam, Fast-Data, parquet, csv

Overview 

Apache Beam is a unified programming model and the name Beam means Batch + strEAM. It is good at processing both batch and streaming data and can be run on different runners, such as Google Dataflow, Apache Spark, and Apache Flink. The Beam programming guide documents on how to develop a pipeline and the ...

Continue reading

Mule on OpenShift: Part 2 - Build & Deploy

Posted by Sohrab Hosseini on 15 March 2019

docker, kubernetes, tech, mule, platform, openshift, container, anypoint

Since the publication of this blog post, MuleSoft has released Mule 4.2. In this version, they have introduced a "start-up performance improvement" that effectively causes a tight coupling between the wrapper and the runtime container. As such, the Unwrapped approach outlined in this post will not work in Mule...

Continue reading

Mule on OpenShift: Part 1 - Deployment Models

Posted by Sohrab Hosseini on 28 February 2019

kubernetes, tech, mule, platform, openshift, container, anypoint

At Deloitte Platform Engineering, we have been building containerised platforms for a few years now. Red Hat OpenShift Container Platform has been our weapon of choice in many of these instances, complementing a solid Kubernetes foundation.

A significant part of this work also consists of building and deploying...

Continue reading

Journey Towards Event-Driven Microservices

Posted by Sudarshan Sreenivasan on 26 November 2018

Events, Microservices, tech, kafka, EDA

Many organizations are adopting a microservices architecture in order to reduce dependencies between system components and allow more frequent release cycles and more flexible scalability. However, unless they have a clear view of the underlying interaction patterns, teams risk building a tightly coupled distributed...

Continue reading

Solving Complex Problems With DataWeave

Posted by Lipika Naik on 16 November 2018

tech, java, mule, dataweave, groovy

DataWeave is the primary transformation language in Mule. What is interesting about DataWeave is that it brings together features of XSLT (mapping), SQL (joinBy, splitBy, orderBy, groupBy, distinctBy operators), Streaming, Functional Programming (use of functions in DataWeave code) to make it a power-packed data...

Continue reading

Comparing Istio with Netflix Frameworks for Inter-Microservices Communications

Posted by Elmira Hasanzadeh on 04 September 2018

Architecture, Microservices, tech, api, istio, netflix, service mesh, gcp

The technical advancements in every aspect of software development lifecycle make it clear that there are more than one solution to any problem. In this article I examine Istio’s service mesh capabilities to address issues that developers face while creating microservices and compare it with the widely adopted Netflix...

Continue reading

AWS Elasticsearch Snapshot Archival for the Brave

Posted by Rahul Sharma on 22 August 2018

Architecture, Logging, tech, aws, audit, archive, elasticsearch

Data retention has become a crucial topic over the past few years, with organisations continuously exploring better ways to define and implement secure processes that cater to their needs. This becomes increasingly difficult when most of our data is stored/generated in the cloud, hence, requiring a flexible and...

Continue reading

Microservices using Spring Boot & Swagger - Part 1

Posted by Bharat Patel on 10 August 2018

Microservices, tech, api, java, swagger, spring

Microservices have become popular as a way of eliminating hidden dependencies between software components and allowing fine-grained deployment without dragging along unneccessary context. In this way, microservices promote autonomy for agile development teams and allow an application to evolve more naturally and in...

Continue reading

OpenShift 3 Demystified. For Developers.

Posted by Sohrab Hosseini on 30 October 2017

tech, developer, platform, openshift, container

Container orchestrators are the next big thing in the modern containerisation movement. And unless you have painted your career into a legacy corner, you will encounter these technologies sooner or later.

OpenShift was my introduction to this space. At that time, I was not able to find suitable online resources that...

Continue reading

Monitoring Anypoint CloudHub API Gateway using Amazon Lambda and ELK

Posted by Nakul Bharade on 26 July 2016

Serverless, tech, elasticsearch, mule, lambda

In a previous blog post, Rashmi showed us how to synchronise logs from CloudHub to an external logging system. This follows an increasing demand to utilise MuleSoft Anypoint CloudHub logs, events, and dashboard statistics as part of a broader monitoring strategy which aims to:

  • Aggregate information in a central...
Continue reading