GitHub Actions – Deploying an Angular App

Recently I built an Angular demo application that showcases some of the features provided by Angular. I will deploy this application to GitHub pages using GitHub Actions, a newly released CI/CD platform that can be used by open source repositories for free.

Since I already have a completed Angular project pushed to GitHub, all I need to do is to create a GitHub workflow to build, test, and deploy the Angular application to GitHub Pages. Before I start, I need to create a folder named .github/workflows at the root of my repository.

To learn more about GitHub workflow, please read workflow syntax for GitHub Actions article.

Create a GitHub Actions Workflow File

In .github/workflows, I added a yaml file for the workflow. And inside the workflow file, you can choose to add the name of your workflow by adding:

name: workflow name

If you omit name inside the workflow file, GitHub will set workflow name to the workflow file path relative to the root of the repository.

GitHub is flexible with however you want to name your workflow file, but the file has to be a yaml file and it has to be in the .github/workflows folder.

Setup Workflow Trigger

A workflow trigger is required for a workflow. I configured the workflow to trigger on pushes to the master branch:

on:
  push:
    branches:
      - 'master'

If you want to use a different trigger for your workflow, please take a look at events that trigger workflows article and on section of workflow syntax for GitHub Actions.

Create the Angular Build And Test Job

In GitHub Actions, jobs are defined by a series of steps that are executed on a runner. Each job runs on a different workspace, meaning that files and job side effects are not kept between jobs. In order to reduce build time and build complexity, I will keep as much work inside one job as possible.

Thus, the job below is created to build and test the Angular application:

jobs:
  build:
    name: Build and Test
    runs-on: ubuntu-latest
    steps: ...

The latest version of Ubuntu GitHub-hosted runner is utilized for this job. But if you want to use a different Github-hosted runner, pease read virtual environments for GitHub-hosted runners article.

Checking out source code

Since jobs do not pull down the source code by default, you need to explicitly tell the job to do so. Therefore, I add the following to steps of build and test job:

- name: Checkout
  uses: actions/checkout@v1

Setup Node.js

To setup Node.js used by the job, add the following under steps of the job:

- name: Use Node 12.x
  uses: actions/setup-node@v1
  with:
    node-version: '12.x'

Build and test job is configured to use Node.js version 12.x. If you wish to use a different version, please take a look at using Node.js with GitHub Actions article.

Run build and test

To build and test the Angular application, I added some supporting scripts to the application’s package.json file:

"build:ci": "ng build --prod --sourceMap=false --base-href /YOUR_REPOSITORY_NAME_HERE/"
"test:ci": "ng test --watch=false --code-coverage --source-map true"

As you can see, the test:ci script will also generate code coverage results for us, which will be used later down the line.

Note: To avoid MIME type error due to invalid path, you need to set your base-href to your repository name

Then, I add the following to the job to build and test our application:

- name: Install dependencies
  run: npm ci
- name: Build
  run: npm run build:ci
- name: Test
  run: npm run test:ci

Upload artifacts

To expose the results of the current job to the next job, I need to configure build and test job to upload the build artifacts. I also configured the job to upload the code coverage results, so they can be reviewed.

- name: Archive build
  if: success()
  uses: actions/upload-artifact@v1
  with:
    name: deploy_dist
    path: dist
- name: Archive code coverage result
  if: success()
  uses: actions/upload-artifact@v1
  with:
    name: deploy_coverage
    path: coverage

if: success() is used to make sure upload artifact only runs if all the previous steps passed. For more information, read context and expression syntax for GitHub Actions article.

Create Deploy Job

With build and test job completed, I can create the job that will deploy the Angular application to GitHub Pages. I add the following yaml below build and test job:

deploy:
  runs-on: ubuntu-latest
  needs: build
  steps:
      - name: Checkout
        uses: actions/checkout@v1
      ...

needs: build is used to tell GitHub to only execute deploy job when build and test job completed successfully.

Download build artifact

I add the following under steps in the deploy job to download build artifact from build and test job:

- name: Download build
  uses: actions/download-artifact@v1
  with:
    name: deploy_dist

To learn more, take a look at persisting workflow data using artifacts article.

Deploy to GitHub Pages

I use GitHub Pages Deploy Action to deploy our Angular build to gh-pages branch of the project repository:

- name: Deploy to GitHub Pages
  uses: JamesIves/github-pages-deploy-action@releases/v3
  with:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    BRANCH: gh-pages
    FOLDER: deploy_dist/YOUR_PROJECT_NAME_HERE

GITHUB_TOKEN is used to avoid providing a personal access token, to learn more about GITHUB_TOKEN, read authenticating with the GITHUB_TOKEN article.

Conclusion

Once you check in your workflow file, which should look similar to the yaml below, to your master branch, you should see a GitHub workflow starting in the GitHub Actions page. When the workflow is complete, you will see the build output and test coverage results in the artifacts section and a branch called gh-pages will be created.

name: workflow name

on:
  push:
    branches:
      - 'master'

jobs:
  build:
    name: Build and Test
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Use Node 12.x
        uses: actions/setup-node@v1
        with:
          node-version: '12.x'
      - name: Install dependencies
        run: npm ci
      - name: Build
        run: npm run build:ci
      - name: Test
        run: npm run test:ci
      - name: Archive build
        if: success()
        uses: actions/upload-artifact@v1
        with:
          name: deploy_dist
          path: dist
      - name: Archive code coverage result
        if: success()
        uses: actions/upload-artifact@v1
        with:
          name: deploy_coverage
          path: coverage
  deploy:
    runs-on: ubuntu-latest
    needs: build
    steps:
      - name: Checkout
        uses: actions/checkout@v1
      - name: Download build
        uses: actions/download-artifact@v1
        with:
          name: deploy_dist
      - name: Deploy to GitHub Pages
        uses: JamesIves/github-pages-deploy-action@releases/v3
        with:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          BRANCH: gh-pages
          FOLDER: deploy_dist/angular-demo

Ensure that your repository has GitHub Pages enabled and the deployment is based off gh-pages branch. If this is set up properly, your Angular application should be deployed!

  • Hannah
  • Sun

A Realist’s Guide to Culture Change

The phrase “we need to change our culture to be successful” has become a punchline for any executive pitching ambitious visions and transformation initiatives, IT-related or otherwise. What is unfortunately less common is any mention of how such a change in culture will happen and how to know when this ideal future culture has been achieved.

Foci is by no means a change management firm nor do I profess any kind of expertise in human behaviour or organizational theory. We are however experts in helping organizations adopt new technologies and methods where culture is an unavoidable challenge. Based on our experience in the trenches, I would like to offer a realist’s perspective on what taking on “culture change” means and what one can expect when committing to this lofty goal.

Culture = People

We can’t talk about culture without defining it first. Culture is an abstraction of what the default behaviours and tendencies of a group of people are. Those behaviours and tendencies are either learned and developed in reaction to how an organization is built and managed, or inherent in the people that are being hired.

When an organization sets out to change its culture, it must accept the reality that it will likely result in a turnover in people. The culture that you desire won’t resonate with everyone in the organization. And a strong culture is built by people who naturally buy into it rather than by trying to hard sell it to someone. Therefore, it’s best to ensure that you are prepared to deal with an increase in turnover and hiring as a part of this commitment rather than assume that a new culture can be achieved without big changes to the workforce.

Culture Requires Nurturing

Executives can’t really dictate the culture of an organization the same way parents can’t really dictate the personalities of their children. An organization’s culture develops based on how people react to and are motivated by that organization’s structure, management style, processes, facilities, compensation model, other employees, and countless other factors. Any attempt to try to define a new culture without looking thoroughly at all aspects of the organization which enabled the current culture would be flawed.

Instead of asking how individuals can adopt the desired behaviours, the organization should ask what aspects of its current structure, policies, compensation, governance, rituals, and general work environment are contributing to the undesirable behaviours and then work to address those. For example, if an organization desires a culture of innovation, budget and approval processes will have to be updated to allow for more experimentation, frequent changes in project parameters, and faster decision making. This is a very organic and fluid process, so set realistic expectations and adapt the plan to how the people are reacting to the changes.

Stress = Negative Behaviours

High stress situations tend to push people to exhibit more basic survival instincts such as territorialism and combativeness. It is extremely difficult for people to adopt more desirable behaviours such as collaboration and transparency or take extra time to think about innovative solutions when timelines are aggressive and budget is tight.

People take time to learn new ways of working and making decisions. This means that efficiency and output will drop before recovering and even improving over the longer term. Project budgets and timelines must account for this and give people enough time to learn the new behaviours and repeat them enough times to become ingrained. It’s the classic “slow down to speed up” adage.

Change Starts at the Top

I am constantly surprised by the number of organizations treating culture change as an exercise whereby the executives look at how they can fix their workforce without also looking at their own behaviour. The culture of an organization is representative of how executives have made decisions over time.

If an organization wants to encourage a culture of taking responsibility, then executives must reflect this by taking actions such as increasing delegation of decision making and making their compensation more outcome-based. If more collaboration is desired, then open door policies must be adopted. Executives can’t just be the champions of change, but also become the examples of the desired culture themselves. The “do as I say and not as I do” philosophy doesn’t work here.

Achieving Success

We are extremely proud of the culture we’ve achieved at Foci. We’ve been deliberate in designing our organization and been very lucky in the type of people we’ve attracted and hired. Here are some of the things that we’ve done and learned about building a strong and innovative culture:

  1. Hire executives with diverse opinions and approaches, but very similar values. Your leadership should have different approaches to solving problems, but should see eye-to-eye on the organization’s core beliefs and philosophies;
  2. Hire for culture fit over pure technical acumen. It’s much easier to teach technical skills than modify behaviours;
  3. Constantly adjust and refine organizational processes and policies. Organizations and the people within them evolve over time. The processes and policies have to be tweaked to account for that;
  4. Create a relationship of mutual trust between our people and the company. Giving people the room to make decisions and exercise judgement encourages a sense of responsibility and ownership. Treat your staff like responsible adults who can make good decisions;
  5. Compensate people based on what you value in your employees. If you want a team that’s constantly upping their game, then compensate for personal growth and skills development;
  6. Invest in people. It’s not just training and some formalized mentorship program. Give people the time, resources, and the infrastructure needed to connect, collaborate, and share knowledge.

Culture change is hard, but by no means impossible. It takes a lot of commitment, attention, investment, time, and patience. By recognizing that the change is really building an organization that nurtures the desirable culture, “we need culture change” will become an achievable call to action rather than just an executive punchline.

  • Shan
  • Gu

Multi Stage Pipelines & Azure DevOps

Many years ago, I wrote a blog post about TFS and DevOps. A lot has changed since then, with multiple versions of the build pipeline being released, but it continues to be one of the most trafficked articles on our site.   Microsoft has worked hard to create a better experience for build automation and continuous integration – so I worked hard on updating this walkthrough. Recently, Microsoft released the idea of multi stage pipelines that work and feel much like how GitLab CI works.

In this post I’ll walk through a basic YAML file and show how you can get a C# project up, built, and tested quickly.

Setup

We need to have a project that is checked into DevOps before we begin. I have a repository that I made for this blog up on DevOps. that is a basic dotnet core console application and a unit test project that goes along with it. At the time of writing this blog post you will also need to turn on the multi-stage pipelines Preview Feature in order to get the best view for these pipelines. You can do that by clicking on the user settings button

User Settings

Then click on preview features

Preview Features

Then ensure that multi-stage pipelines are enabled

Multi stage pipelines enabled

First Steps

First we need to add a YAML file into our project. I tend to put this file directly at root and name it azure-pipelines.yaml. Then we need to define our stages. A stage is a collection of jobs and can be run concurrently or can be dependent on another stage successfully completing. For this quick project we will have two different stages

  • Build
  • Test

In order to define these stages in our pipeline we need to write some YAML like

stages:
  - stage: build
    displayName: Build
  - stage: test
    displayName: Test
    dependsOn:
    - build

this will give us building blocks to add our jobs. If you check this file into DevOps and navigate to pipelines you can see that we have a pipeline defined without any runs associated to it.

multi stage pipeline showing in dashboard

Adding a Job

A job runs on a build agent. By default DevOps provides hosted build agents. These agents are a pre-configured VM that have a lot of different development tools pre-installed. I’ll be using the hosted agents for this post.

Let’s add in some YAML to add a job that will build our dotnet solution. We can do this in one of two ways, we can use a DevOps “task” or we can write a script. Tasks can provide a lot of features that you would normally need to script yourself. These can be very helpful, however it also hides a lot of what is being run. I tend to try and use tasks as they get updated regularly to add additional features and fix bugs. Microsoft hasn’t made tasks to solve every problem however so you will need to write some scripts eventually.

Example as Task

variables:
  buildConfiguration: "Release"
  
stages:
- stage: build
  displayName: Build
  pool:
    vmImage: "Ubuntu 16.04"    
  jobs:
  - job: build_dotnet_solution
    displayName: build dotnet solution
    steps:
    - task: DotNetCoreCLI@2
      inputs:
        command: build
        arguments: '--configuration $(buildConfiguration)'
- stage: test
  displayName: Test
  dependsOn:
  - build

Example as script

variables:
  buildConfiguration: "Release"
  
stages:
- stage: build
  displayName: Build
  pool:
    vmImage: "Ubuntu 16.04"    
  jobs:
  - job: build_dotnet_solution
    displayName: build dotnet solution
    steps:
    - script: |
      dotnet build --configuration $(buildConfiguration)
- stage: test
  displayName: Test
  dependsOn:
  - build

In both examples I have added a variable to set the build configuration setting for the pipeline. Variables are very helpful and DevOps also provides a lot of pre-defined variables for you. You can ready about them here.

Artifacts

Now that we have our job running and our solution is being built. We will probably want to retain these files. We will need to artifact these files if we want to use them in a different job, or we can download them later for manually testing the build.

variables:
  buildConfiguration: "Release"
  
stages:
- stage: build
  displayName: Build
  pool:
    vmImage: "Ubuntu 16.04"    
  jobs:
  - job: build_dotnet_solution
    displayName: build dotnet solution
    steps:
    - task: DotNetCoreCLI@2
      inputs:
        command: build
        arguments: '--configuration $(buildConfiguration)'
    - publish: $(System.DefaultWorkingDirectory)/src/demo-project/bin/$(buildConfiguration)/netcoreapp3.0/
      artifact: source
- stage: test
  displayName: Test
  dependsOn:
  - build

Once the build is completed you should see the artifacts on the build page. You can download them and use them in different jobs now.

multi stage pipeline artifacts published

Testing

Now that we have our code built, we can go ahead and run the tests for our application. DevOps also has the ability to show us test results through its dashboards. It’s easiest to use the task for this, as the task has capabilities to upload the tests results for us.

variables:
  buildConfiguration: "Release"
  
stages:
- stage: build
  displayName: Build
  pool:
    vmImage: "Ubuntu 16.04"    
  jobs:
  - job: build_dotnet_solution
    displayName: build dotnet solution
    steps:
    - task: DotNetCoreCLI@2
      inputs:
        command: build
        arguments: '--configuration $(buildConfiguration)'
    - publish: $(System.DefaultWorkingDirectory)/src/demo-project/bin/$(buildConfiguration)/netcoreapp3.0/
      artifact: source
- stage: test
  displayName: Test
  dependsOn:
  - build
  jobs:
  - job: test_dotnet_solution
    displayName: test dotnet solution
    steps:
    - task: DotNetCoreCLI@2
      inputs:
        command: test        
        arguments: '--configuration $(buildConfiguration)'
multi stage pipeline tests successful

With this, you now have a basic build and test pipeline that will run with every check-in to your repository. There is a lot more that can be done, such as managing environments and performing releases. I hope that this is a good starting block to get you moving with DevOps.

  • Dan
  • McCrady

What’s Your Organization’s Rocket Fuel?

The conversations I regularly have with clients, other executives, and my mentors are usually around “what’s your org’s vision?” or “what do you want your org to do”. Foci has gone through a tremendous period of growth and change over the last 12 months and the answers to those questions seem to be ever changing. This has led me and the rest of the management team to have some very interesting discussions around how we define Foci and our purpose.

The “what” and the “how” doesn’t matter

Photo by John Baker on Unsplash

Regardless of how well thought out your vision or strategy is, the reality is that $%*@ happens. Your clients can change their mind, you may lose some key contracts, the market will evolve and change, competitors will emerge, or you may not be able to get that unicorn architect/developer to run that world-changing product you want to build. And every time you have to make a pivot to adjust to those changes, it can be a very painful experience, both for you and your team.

The identity of an organization is very important, especially if you have a strong team culture like us. Team members imprint themselves onto that identity and subconsciously use it as a reference point for their everyday work. We started life as an Oracle Fusion Middleware company, then became an Architecture and Integration company, and now we’re doing more Cloud-Native custom dev with a broader range of system integration and program management services. Each of those shifts in focus created quite a bit of disruption in the team. People asked “Wait what? I thought we were doing the other thing? What does that mean to our existing projects? Will we stop doing the other thing altogether?” These were all fair questions, but after working through it all, we noticed that none of it actually impacted our team culture or our core behaviours.

What we took away from this were 2 things:

  1. What you did as a firm or how you did it had very little alignment to your culture.
  2. Our people are very emotionally connected to Foci’s identity and feel any change in that identity keenly.

It’s all about the “why”

This naturally led us to look at why our folks joined Foci and what made them excited about coming into work each day. Turns out no one was really driven by the prospect of writing thousands of lines of C# code, installing and configuring Oracle SOA Suite, or creating a stakeholder engagement plan. Sure, those things interested people, but they weren’t really core motivators.

We ended up landing on 2 aspects of motivation that were the most important:

  1. What brings you the most satisfaction (e.g., solving a problem, having an impact, getting recognition, seeing something you’ve built be used)
  2. What is your metric of value (e.g., complexity of the problem, number of people impacted, transaction volume, financial savings)

Problems are our rocket fuel

We always joked about having a generic tagline like “we solve problems” (it’s on our website) because we were constantly evolving the business. Appropriately that turned out to be the answer. What we realized is that our entire team and our hiring processes all coalesced around the core desire to solve complex and interesting problems. We weren’t motivated by how many people were using an app that we had developed or whether the systems we helped our clients build were processing 100 or 1,000,000 transactions a day.

The thing that gave us all a real sense of accomplishment and gave us that little shot of dopamine we humans naturally crave was when we were able to solve a problem for our client. The bigger the complexity greater the satisfaction. As long as we had a healthy supply of complex and interesting problems to feed our team, we could go anywhere.

The destination and the things you do to get there will always change over time. The things that motivate and drive you to move forward are more constant and core to your being. Defining your organization based on the goal you want to achieve or the tasks that you do makes every pivot feel like an identity crisis. Putting in the time to identify the rocket fuel that constantly propels your team forward creates a solid corporate identity to anchor against regardless of the path your organization decides to take. Interesting problems are our rocket fuel. As long as we as a management team ensure that our team has a steady flow of interesting problems to solve, we can have every confidence in Foci’s ability to achieve any goal that we set for ourselves. Until we change our mind, of course.

  • Shan
  • Gu

The Digital Transformation Road Trip

Too often, we see articles shared preaching the importance for organizations to adopt a digital strategy without encapsulating what that really means. To remove some of that confusion, I like comparing an organization’s digital transformation with something everyone knows – a road trip!

Let’s start with some truth – a digital strategy can be enormously beneficial to a department or organization and its ability to deliver value to customers. BUT, like a good road trip, becoming a digital organization isn’t an overnight journey – and it doesn’t always follow a set path. It requires planning, understanding, commitment, and the ability to embrace the detours along the way.

Oh the places you’ll go!

Before setting out on a trip, you need to have a destination in mind.  Similarly, executives need to agree on what ‘digital’ means for their organization. What problems are you really trying to solve?  Once you have these identified, your organization can begin to evaluate the possible ways of getting there.  

Loading up the car

Digital transformation is as much a business transformation as an IT one. Digital processes are about re-examining your business from top to bottom in order to have the right information, available at the right time, to make the right decisions. Cooperation, communication, and most importantly – organization-wide understanding is key to making sure this happens.

It’s important to start with the problem without focusing on what the solution might end up being. Challenge pre-existing assumptions and ideas about who should be doing what, when, and how. Break down your tools and processes so that you can rebuild it in a more efficient, modern way.

Digital organizations have governance and management frameworks that are very different than paper-based organizations. Keeping everyone involved ensures that you have multiple sets of eyes on the road. Making sure they know why this journey is happening means they’re looking out for the right kind of obstacles and opportunities.  

Take advantage of the bumps along the way

A digital organization embraces speed, communication, learning, and also, failure. It’s less important to have a map setting the route in stone from  start to finish than it is to be aware of what’s going on around you. Being aware of your surroundings lets you be prepared to change direction when a better path becomes available (or to avoid that head-on collision up ahead)! A digital organization uses this awareness to stay relevant and ahead of the curve. Approaches and methods like incremental development, democratized governance, Test Driven Development, and Agile are all designed to support teams in this way.

This can be a big change in thinking, especially for larger, more traditional organizations. Understanding which tools are available, and when to leverage them, can significantly improve your chances of finding success in your transformation.

Embrace being off the beaten path

So – before embarking on your digital journey, make sure you understand where it is you want your organization to go, focus on the journey, and be prepared to embrace being off the beaten path. You might not take the path you first imagined, but digital transformation is about the journey, not the destination.

  • Kevin
  • Steele

Why .NET loves Linux

This is an update to the post I made a while ago titled “Why .Net Doesn’t need to be Expensive“.  A lot has changed since I made that post.  For example: Visual Studio Code wasn’t released, .NET Core was called vNext, and .NET hadn’t gone through it’s open-source transformation.  These introductions to the .NET ecosystem have changed the way .NET developers are working day-to-day and the path to deploying .NET on Linux is quickly becoming a mandatory requirement for IT shops.

Microsoft has been on the Linux love train for quite some time now, and we are slowly starting to see the fruits of this transformation.  Just recently the Linux Subsystem on Windows was added to Windows 10 without the need to turn on developer mode.  Developers now have a native Linux bash that can be enabled through the Windows store.  The new .NET core project templates in Visual Studio include Docker support with just the click of a checkbox.  Azure allows you to host .NET Core projects in Linux, and has moved to allow container orchestration using technologies like Azure Container Storage, and soon to come Azure AKS (its managed Kubernetes).  This change is also reaching out to the open source community.  Most large projects have either ported their program to use .NET standard or are in the process of converting it.

 

Why so much Love?

Plain and simple: moving custom code to the cloud means moving to Linux.  All cloud technologies that are coming out have Linux as a common player.  AWS, GCP, OCP, Azure, and even smaller players like Digital Ocean all provide Linux as the OS.  If an IT organization can’t migrate their .NET custom code to Linux they are dramatically limiting the choices they have to get to the cloud.  If you aren’t going with Linux you only have two real choices:

1)  Find a Windows Server VM in the cloud and deploy to IIS.  

Technically yes, you are moving to the cloud, but are you really gaining any benefits?  Your operations team still needs to monitor, maintain, and patch this VM just as if it was in your private data centre.  You also are quickly locking yourself to the provider since making an export of the VM to move to another provider will be difficult and require down time as you make that transition.

2)  Use Azure PaaS Offerings like Web App Services.  

Azure is still your friend here.  They will take your web application code that is slightly modified to be cloud ready and host it for you.  The Web App Services offering is really good stuff.  It comes with free auto-scaling, monitoring, and guaranteed availability.  They even take care of patching and maintaining the infrastructure.  The downside here is that until you have migrated that application to Linux you are tied to Azure.  No other cloud provider is looking at a way to host non-core .NET web sites.  So if Azure changes the pricing model, you will need to change with it.

 

What does Linux get you?

Linux buys you true portability of your applications. The most common way to get to true application portability is to write your applications as a 12 factor application, while using Docker to wrap your application and prepare it for deployment.  If you follow this procedure, then pretty much any platform is open for you to deploy your applications.  Microsoft is currently working to create Windows Server Docker containers like microsoft/nanoserver, but the licensing and deployment constraints are still unclear.  It appears that you need to deploy these images only on a licensed Windows Server 2016 system.  This restriction makes your application tightly coupled to Windows systems and reduces your deployment options significantly.

 

More investment for .NET Developers

A little while ago I was talking to a group about how the shift to Linux will be a big shift for .NET developers. Normally Microsoft would have a major release and developers could focus for a year or so to wrap their heads around it.  When the TPL was released, Async Await was the big player. Bloggers would write endless articles on how leverage this feature to introduce multi-threading into applications.  This update was all that .NET developers needed to focus on.  The next few years are changing a lot more than Async Await.  A new Operating System in Linux, arguably a new framework with .NET Core, Docker containers, container orchestrators like Kubernetes, all while building strong Dev Ops capabilities.  The future is bright for .NET but the time required to learn all the advantages is long.  I plan to keep our developers moving in this direction, since it is the brightest path forward for custom software development in general, including the .NET ecosystem.

 

  • Dan
  • McCrady

Using JavaScript and JSON as a Common Language in Orbital Bus

Large enterprises usually have many programming languages across their departments. These departments, often located in different cities, will build teams out of what they see as the best-available local resources. It’s fairly common to find large-scale enterprise or government groups that have applications written in .NET and Java, never mind the plethora of other languages and flavours thereof. This technological mishmash is a major challenge to any sort of enterprise service bus; one that Orbital Bus is trying to overcome.

In creating Orbital Bus, we decided at the start that developers shouldn’t have to learn any new languages to implement our solution. The learning curve had to be minimal to ensure wide-spread adoption. We were able to deliver some of that goal by creating our Code Generation utility. This tool would allow us to take a single input and compile it to code usable by our ESB. However, this tool still needs input, so what were we to do?

Enter Javascript. We decided that by making the code generation input Javascript we would make it accessible to as many developers as possible with no extra work. No matter what language you develop in, you’ve probably had to work on some Javascript, whether to create visual effects or to load data with an Ajax call. We could implement Javascript with a high degree of confidence that users would be able to work with it without any sort of intimidating ramp. Javascript also provides a feature-rich environment that we don’t have to worry about maintaining. If developers want functionality that already exists in a library it’s minimal work for them to implement it. Along with Javascript, we were also able to rely on the JSON schema standard for modelling objects. We don’t have to worry about maintaining an API for describing models in our system. We simply have to point towards the standard we support and let the JSON schema community do the heavy lifting.

What exactly are we doing with all this Javascript? I mentioned the use of schemas to define models. We use models to define the types which are expected for the receiver. We take in standard JSON schemas to create C# classes which are then deployed as part of a contract library with the receiver. This library is used by receiver and the dispatcher. (Check out our article about using MEF with our contact libraries.) The models defined in this schema are also the ones expected by our translation engine. The receiver node of Orbital Bus takes Javascript translation files which it executes in both directions. With this feature developers can implement any translation they want as the information passes through the receiver node. These translations are simple .js files with method calls. We even support logging and business errors through integrated methods. Check out our documentation for more information on implementation. We even used JSON files for our configurations rather than XML to make sure that our points of contact with Orbital Bus are as unified as possible. As we grow Orbital Bus’ functionality we expect to grow its use of Javascript.

The default Javascript translation template.
The default Javascript translation template.

It was tough trying to think of the best way to support a polylinguistic development environment. Thankfully Javascript gave us a single point of entry we could use across many development environments. There’s still work we want to do with our Javascript implementation. We want to integrate libraries by default in our translations, allowing developers to use library calls without having to include them manually. We also want to add Javascript to our collection of connectors for the Orbital Bus. Thankfully, with a common input set out, Orbital Bus will be free to grow its implementations while continuing to support developers from a wide variety of backgrounds.

  • Joseph
  • Pound

Dynamic Plugin Loading Using MEF

The Managed Extensibility Framework (MEF) is a library that enables software to discover and load libraries at runtime without hard-coded references. Microsoft included MEF in .NET framework version 4.0 and since then it has been commonly used for dependency resolution and inversion of control patterns.

Orbital Bus makes communication possible between different parties by sharing contract and schemas. A receiver has a contract library that has all the information needed for a dispatcher to make proper synchronous and asynchronous calls all the way to an end consumer. The dispatcher downloads a receiver’s contract library and then uses it to construct calls with the right data schemas. It became very clear to us during development that a crucial requirement was that the dispatcher to be able handle any downloaded contract library DLL and process it without code changes. This is where MEF comes into play. It lets us inject libraries, in this case the receiver’s contract libraries, at the start-up stage.

Once we chose to use MEF as our integration tool, we were able to start the Code Generation Project. This project is a convenient CLI tool that efficiently generates the contract libraries and plugins which are loaded by the receiver. These libraries are made available for download to any dispatcher on the mesh network. One challenge we encountered downloading multiple contract libraries for the dispatcher was how to distinguish between these contract libraries. What if two contracts have similar operation names? How can the dispatcher tell what is the right operation to select from its composition container? We were able to solve this challenge by making sure that each contract library generated has a unique ServiceId that would be exported as metadata within the contract library. This setting enables the dispatcher to filter out various operations based on their ServiceId:

    namespace ConsumerContractLibrary
    {
        [ExportMetadata("ServiceId", "ConsumerLibrary")]
        public class AddCustomerOperation : IOperationDescription {}
    }

When the receiver starts up, it will pull the plugins from its Plugins folder and load the plugin.dll and adapters into MEF’s CompositionContainer, a component used to manage the composition of parts. Those dependencies will be injected into the receiver as it loads. In addition to handling messages destined for the consumer, the receiver also serves as file server that waits for the dispatcher to download the contract library when needed.

    public PluginLoader(IConfigurationService config)
    {
        this.config = config;
        var container = this.BuildContainer(); // load the plugin DLLs and create composition container
        this.RegisterAdapters(container);
        var details = this.RegisterPlugins(container);
        this.BootStrapSubscriberDetails(details); //Creates needed dependencies and bootstraps the given details.
    }

After a dispatcher downloads the available contract library specifications into a composition container, it will filter out and return all the exported values in the container corresponding the given ServiceId.

    public static IEnumerable<T> GetExportedValues<T>(this CompositionContainer container,
            Func<IDictionary<string, object>, bool> predicate)
    {
        var exportedValues = new List<T>();

        foreach (var part in container.Catalog.Parts)
        {
            foreach (var ExportDef in part.ExportDefinitions)
            {
                if (ExportDef.ContractName == typeof(T).FullName)
                {
                    if (predicate(ExportDef.Metadata))
                        exportedValues.Add((T)part.CreatePart().GetExportedValue(ExportDef));
                }
            }
        }

        return exportedValues;
    }

Where the predicate clause is actively the filter we need for ServiceId:

    metadata => metadata.ContainsKeyWithValue(METADATAKEY, serviceId)

After filtering the process, the dispatcher has all the contract library operations that are supported by the receiver.

MEF proved invaluable in solving the problem of runtime library integrations and to enable the plugin architecture. This implementation allows Orbital Bus the flexibility for developers to customize or update their contract libraries, service configurations, and translations without affecting other services on the bus. As our work continues, we plan on looking closer at the issue of versioning in the dispatcher to keep its cache in sync with the receiver’s contract libraries, making Orbital Bus an even more agile messaging solution.

  • Dan
  • McCrady

Continuous Integration: Balancing Value and Effort

Continuous integration can be a tough sell to managers. It’s hard to describe the need for extra time and resources to build automated tests that should mimic what is already being done by developers. This advocacy can be especially difficult early in development when CI failures are common and the pipeline will need a lot of work. Why would any manager want a tool that creates more problems and interferes with the development cycle? A robust continuous integration pipeline is vital during development since it protects from the deployment of broken code and will generate more issues to remove bugs before production. Since Orbital Bus is an internal project, we decided to use it as an opportunity to build the kind of CI pipeline we always wanted to have on client sites.

Early on we looked at the possibility of automated provisioning of multiple machines for integration tests. We looked at a variety of tools including Vagrant, Salt Stack, and Chef and Puppet. What we found is that this automation was not worth the time investment. This post is supposed to be about the value of investing in a CI pipeline, so why are we talking about work we abandoned? To demonstrate that the value of a CI pipeline has to be proportionate to the time cost of maintaining it. When it came to automated provisioning we realized that we would spend more time maintaining that portion of the pipeline than reaping the benefits, so we stood up the VMs manually and replaced provisioning with a stage to clean the machines between runs.

As development progressed, we added to our pipeline, making sure that the time investment for each step was proportionate to the benefits we were receiving. Gradually we added the build process, unit tests, and automated end-to-end integration tests. As we continued to experiment we began using the GitLab CI runners to enhance our testing. We also discovered that GitLab could integrate with Jenkins, and brought our pipelines together to create an integrated dashboard on GitLab. As we neared the public release, we added a whole new stage for GitLab pages to deploy our documentation.

A shot of our Jenkins Continuous Integration pipeline builds.
A shot of our Jenkins pipeline builds.

As the saying goes, Rome was not built in a day. Neither was our continuous integration. We added to it gradually, and as we did we had to overcome a number of obstacles. Our greatest problem has been false negatives. False negatives immediately negate the benefits of continuous integration because the team stops respecting the errors being thrown by the system. At one point, our disregard for the failures on the CI pipeline prevented us from noticing a significant compatibility error in our code. Each failure was an opportunity for us to understand how our code was running on multiple platforms, to explore the delta between development and production environments, and ultimately made our solution more robust. From the perspective of productivity it was costly, but the time greatly outweighed the value of hardening of our solution.

A capture of one of our Continuous Integration GitLab pipelines.
A capture of one of our GitLab pipelines.

You would be mistaken if you thought we’ve stopped working on our pipeline. We have plans to continue to grow our CI, expanding our integration tests to include performance benchmarks and to work with the multiple projects which have originated in the Orbital Bus development. These additional steps and tests will be developed alongside our new features, so as to integrate organically. As our solution matures, so will our continuous integration, which means we can continue to depend on it for increased returns in our development cycle.

  • Joseph
  • Pound

Securing the Orbital Bus

After getting familiar with the Orbital Bus Architecture, and how it solves the traditional Enterprise Service Bus (ESB) shortcomings, it was time for our development team to create a secure solution around the components and communication channels of our distributed solution.

The challenge:

The Orbital Bus makes exchanging information possible between various parties. To this effect, Orbital Bus has three components involving communication:

  • RabbitMQ is the message broker across its various components.
  • Orbital Bus service registration and discovery is built on top of Consul.
  • The receiver calls out to the consumer with a REST Adapter that is built using the RestSharp Client.

These communication components support TLS-Encryption and HTTP authentication. We also want to support additional authentication and message protection mechanisms in the future. In order to implement these solutions Orbital Bus needs to provide a way to save credentials, X.509 certificates, and other forms of tokens. To summarize, the challenges we encountered in developing Orbital Bus were:

  1. Provide a secure vault to store various types of credentials, certificates, and tokens.
  2. Make the security features optional so it could be implemented only when needed.

The Solution

While working on the Orbital Bus it became obvious that a secure vault was needed to save sensitive information such as credentials, tokens, and certificates. Inspired by Java Keystore, Foci Solutions designed and developed a platform-agonstic C# Keystore solution that could work on Windows or Linux. Foci’s Keystore is available as a Nuget Package, and it also comes with a Keystore Manager CLI Tool to perform CRUD operations on the Keystore directly. Please visit the Keystore’s How to Guide for more details on how to use the Keystore and its manager.

The Keystore addresses the first security challenge. Your system requires a secure RabbitMQ client? Not a problem. You can have the credentials saved in the Keystore and use them whenever needed. Your Orbital Bus implementation requires using a certificate for service discovery through Consul? The Keystore can encrypt and save the certificate to be used whenever needed. If you look closely at the Orbital Bus API Documentation, you will notice that there is a KeystoreService and a KeystoreRepository that makes the integration with Foci’s Keystore Seamless. The Keystore’s CRUD repository makes it available to any part of the Orbital Bus components via the KeystoreService.

Now that the first security challenge has been addressed through the Keystore integration, let’s move on to the second challenge: How to make security available but optional? The first thought that comes to mind is to modify Orbital Bus code. After further consideration, it becomes very clear that code modification based on security requirements is an expensive approach that necessitates code change based on the implementation requirements. We decided to integrate the security options into our configuration service to allow changes on the fly. This way security options throughout the Orbital Bus solution can be toggled with minimal effort. You want to secure your dispatcher’s communication to the RabbitMQ client? Then all you need is to turn on a security flag and provide the RabbitMQ credentials. Just let Orbital Bus’ configuration service take care of the rest.

How to Use the Keystore

Foci’s Keystore can accommodate various entry types like certificates, key pairs, username/password pairs, and tokens. Each entry in the Keystore has a unique Alias to keep them organized. The Keystore can be configured to encrypt/decrypt its content using either the Current User or Local Machine data protection scopes. The Keystore is fully integrated for use by any component of the Orbital Bus like the dispatcher or receiver. You will only need to initialize the Keystore with the Keystore Manager Tool and add any credentials or certificates your solution requires. For example: Your implementation requires a secure communication between the dispatcher and RabbitMQ using a username and password? All you need to do is create the Keystore using the Keystore Manager Tool and add a new entry for the required credentials with a unique Alias. What’s next? How to retrieve the stored entries? What’s this Alias for? How to use it? All this will be explained in the next section.

How to Configure Security

Orbital Bus approach favours configuration over customization for obvious reasons. In this section we will walk through how you can configure RabbitMQ, Consul, and the REST Adapter to be secure. The Orbital Bus has a KeystoreService that sits in the Root of the solution. The KeystoreService is injected into the ConfigurationService class. This ConfigurationService is a powerful and flexible tool. It can be injected into any component and it imports any set of configurations that are stored in a specified JSON file mapped into their own configuration model. For example: The ConfigurationService is injected into the DispatcherRegistration in order to configure the dispatcher with settings including the RabbitMQ options for addresses, credentials, and certificates.

RabbitMQ Configuration

Both the dispatcher and receiver establish RabbitMQ buses that can be configured as secure. The following is a JSON configuration file for a dispatcher that has the RabbitMQ security enabled:

{
  "DispatcherConfiguration":
  {
    "BusHostIp": "localhost",
    "BusHostPort": 5672,
    "ContractLibraryPrefix": "",
    "ConsulConfiguration": {
      "HostIp": "localhost",
      "HostPort": 8500
    }
  },
  "BaseConfiguration": {
    "RabbitMQConfiguration": {
      "BusHostIp": "localhost",
      "BusHostPort": 5672,
      "SslEnabled": "true",
      "Alias": "rabbitCredentials"
    }
  }
}

You might notice that the property SslEnabled is set to true, and there is an Alias property with the value “rabbitCredentials”. This simple configuration allows Orbital Bus to enable secure communications with the RabbitMQ server. The Alias here is the unique name we assigned to the credentials entry saved in the Keystore using the Keystore Manger Tool. Securing RabbitMQ in Orbital Bus is as simple as this. Save your credentials in the Keystore, and make sure you edit your configuration to point to the stored credentials Alias.

Consul Configuration

For Orbital Bus we implemented Consul security connections with certificate authentication. Any REST client or request created to communicate to Consul should have an appended certificate for authentication. In return, Consul will return its certificate to authenticate to the client. The following is a JSON configuration file for a dispatcher that has the Consul security enabled:

{
  "DispatcherConfiguration":   
  {
    "BusHostIp": "localhost",
    "BusHostPort": 5672,
    "ContractLibraryPrefix": "",
    "ConsulConfiguration": {
      "HostIp": "localhost",
      "HostPort": 8500
    }
  },
  "BaseConfiguration": {
    "ConsulConfiguration": {
      "HostIp": "localhost",
      "HostPort": 8500,
      "SslEnabledConsul": "true",
      "Alias": "consulcert"
    }
  }
}

Here a similar approach to the RabbitMQ implementation is used. An entry with the Alias “consulcert” is referenced to retrieve the stored certificate that would be injected into the ConsulService when its initialized. The service then appends that certificate to requests.

REST API

The REST Adapter follows a similar approach to enable and configure secure HTTP communications. The RestAdapterConfiguration class has a SecureConsumer flag to indicate if the security is enabled and a ConsumerAlias contains the unique Alias name for the credentials in the Keystore.

Security is always a pressing concern and the best solution is not often easily apparent. In building the Keystore, we sought to make a tool that could be used easily and repeatedly, while at the same time making it an integral part of the Orbital Bus. We recommend checking out the How To Guide and trying it out yourself.

  • Rabie
  • Almatarneh