Tag: Orbital

Using JavaScript and JSON as a Common Language in Orbital Bus

Large enterprises usually have many programming languages across their departments. These departments, often located in different cities, will build teams out of what they see as the best-available local resources. It’s fairly common to find large-scale enterprise or government groups that have applications written in .NET and Java, never mind the plethora of other languages and flavours thereof. This technological mishmash is a major challenge to any sort of enterprise service bus; one that Orbital Bus is trying to overcome.

In creating Orbital Bus, we decided at the start that developers shouldn’t have to learn any new languages to implement our solution. The learning curve had to be minimal to ensure wide-spread adoption. We were able to deliver some of that goal by creating our Code Generation utility. This tool would allow us to take a single input and compile it to code usable by our ESB. However, this tool still needs input, so what were we to do?

Enter Javascript. We decided that by making the code generation input Javascript we would make it accessible to as many developers as possible with no extra work. No matter what language you develop in, you’ve probably had to work on some Javascript, whether to create visual effects or to load data with an Ajax call. We could implement Javascript with a high degree of confidence that users would be able to work with it without any sort of intimidating ramp. Javascript also provides a feature-rich environment that we don’t have to worry about maintaining. If developers want functionality that already exists in a library it’s minimal work for them to implement it. Along with Javascript, we were also able to rely on the JSON schema standard for modelling objects. We don’t have to worry about maintaining an API for describing models in our system. We simply have to point towards the standard we support and let the JSON schema community do the heavy lifting.

What exactly are we doing with all this Javascript? I mentioned the use of schemas to define models. We use models to define the types which are expected for the receiver. We take in standard JSON schemas to create C# classes which are then deployed as part of a contract library with the receiver. This library is used by receiver and the dispatcher. (Check out our article about using MEF with our contact libraries.) The models defined in this schema are also the ones expected by our translation engine. The receiver node of Orbital Bus takes Javascript translation files which it executes in both directions. With this feature developers can implement any translation they want as the information passes through the receiver node. These translations are simple .js files with method calls. We even support logging and business errors through integrated methods. Check out our documentation for more information on implementation. We even used JSON files for our configurations rather than XML to make sure that our points of contact with Orbital Bus are as unified as possible. As we grow Orbital Bus’ functionality we expect to grow its use of Javascript.

The default Javascript translation template.
The default Javascript translation template.

It was tough trying to think of the best way to support a polylinguistic development environment. Thankfully Javascript gave us a single point of entry we could use across many development environments. There’s still work we want to do with our Javascript implementation. We want to integrate libraries by default in our translations, allowing developers to use library calls without having to include them manually. We also want to add Javascript to our collection of connectors for the Orbital Bus. Thankfully, with a common input set out, Orbital Bus will be free to grow its implementations while continuing to support developers from a wide variety of backgrounds.

  • Joseph
  • Pound

Continuous Integration: Balancing Value and Effort

Continuous integration can be a tough sell to managers. It’s hard to describe the need for extra time and resources to build automated tests that should mimic what is already being done by developers. This advocacy can be especially difficult early in development when CI failures are common and the pipeline will need a lot of work. Why would any manager want a tool that creates more problems and interferes with the development cycle? A robust continuous integration pipeline is vital during development since it protects from the deployment of broken code and will generate more issues to remove bugs before production. Since Orbital Bus is an internal project, we decided to use it as an opportunity to build the kind of CI pipeline we always wanted to have on client sites.

Early on we looked at the possibility of automated provisioning of multiple machines for integration tests. We looked at a variety of tools including Vagrant, Salt Stack, and Chef and Puppet. What we found is that this automation was not worth the time investment. This post is supposed to be about the value of investing in a CI pipeline, so why are we talking about work we abandoned? To demonstrate that the value of a CI pipeline has to be proportionate to the time cost of maintaining it. When it came to automated provisioning we realized that we would spend more time maintaining that portion of the pipeline than reaping the benefits, so we stood up the VMs manually and replaced provisioning with a stage to clean the machines between runs.

As development progressed, we added to our pipeline, making sure that the time investment for each step was proportionate to the benefits we were receiving. Gradually we added the build process, unit tests, and automated end-to-end integration tests. As we continued to experiment we began using the GitLab CI runners to enhance our testing. We also discovered that GitLab could integrate with Jenkins, and brought our pipelines together to create an integrated dashboard on GitLab. As we neared the public release, we added a whole new stage for GitLab pages to deploy our documentation.

A shot of our Jenkins Continuous Integration pipeline builds.
A shot of our Jenkins pipeline builds.

As the saying goes, Rome was not built in a day. Neither was our continuous integration. We added to it gradually, and as we did we had to overcome a number of obstacles. Our greatest problem has been false negatives. False negatives immediately negate the benefits of continuous integration because the team stops respecting the errors being thrown by the system. At one point, our disregard for the failures on the CI pipeline prevented us from noticing a significant compatibility error in our code. Each failure was an opportunity for us to understand how our code was running on multiple platforms, to explore the delta between development and production environments, and ultimately made our solution more robust. From the perspective of productivity it was costly, but the time greatly outweighed the value of hardening of our solution.

A capture of one of our Continuous Integration GitLab pipelines.
A capture of one of our GitLab pipelines.

You would be mistaken if you thought we’ve stopped working on our pipeline. We have plans to continue to grow our CI, expanding our integration tests to include performance benchmarks and to work with the multiple projects which have originated in the Orbital Bus development. These additional steps and tests will be developed alongside our new features, so as to integrate organically. As our solution matures, so will our continuous integration, which means we can continue to depend on it for increased returns in our development cycle.

  • Joseph
  • Pound

Testing an ESB with Selenium!

Manual integration testing is the first “test plan” in application development. This strategy quickly becomes a pain, so introducing automation early is important with any project. When working on an enterprise service bus, automated testing is a must have. Early in the development of Orbital Bus we began implementing automated unit tests and even automated integration tests in our continuous integration pipeline. Scripting such tests is a common approach for similar ESB projects. Why are we writing an article about Selenium then? Eventually we decided that console applications weren’t enough. It’s very common to have web applications that call to multiple micro-services via an ESB, so we wanted to build one and test it out. How would we automate those tests? We chose Selenium.

 

Selenium is an open-source testing solution for web apps. Selenium helps you automate tasks to make sure your web application is working as you expect. Selenium has multiple tools:

  • Selenium 2 (WebDriver): Supplies a well-designed object-oriented API that provides improved support for modern, advanced web-app testing problems.
  • Selenium 1 (Remote Control): Provides some features that may not be available in Selenium 2 for a while, including support for almost every browser and support of several additional languages (i.e. Java, Javascript, Ruby, PHP, Python, Perl, and C#).
  • Selenium IDE: Helps with rapid prototyping of tests with Selenium for experienced developers or beginner programmers who are looking to learn test automation.
  • Selenium-Grid: Allows for tests to run on different machines and in different browsers in parallel.

For Orbital Bus, we used Selenium 2 since it supports nine different drivers (including mobile OS’s). That ensures going forward we can adapt our tests however we need.

We know Orbital Bus will be an essential component in a web application, so we needed to test the ability of Orbital Bus to handle a lot of requests from a browser-based application. By stressing Orbital Bus with a lot of requests, we could verify that our message broker receives and delivers the messages properly and the Dispatcher and Receiver handles and translates the messages as expected.

In addition to Selenium 2, we needed to use a web driver to conduct the testing. We chose the Phantomjs Driver. This driver is “headless”, meaning we were able to manipulate all the elements in the page. Keeping in mind we wanted to stress test the system, we weren’t concerned with complicated UI scenarios. We were using Selenium just to fire requests and make sure that a UI element changed showing receipt of the response. Other test harnesses for browsers often open extra pages to capture the screen. That kind of interaction was beyond our scope. We wanted to just focus on messages getting through. The following code section is an example of our tests.


public void RequestsForYahooWeather()
{

var countryJsonPath = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "Configs", "countries.json");
var jsonString = File.ReadAllText(countryJsonPath);
List countries = JsonConvert.DeserializeObject<List>(jsonString);

foreach (country countryO in countries)
{
var driver = new PhantomJSDriver();
WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10));
driver.Navigate().GoToUrl("https://localhost:55794/Home/Weather");
IWebElement wElement = driver.FindElement(By.Id("CityName"));
IWebElement wElementTemperature = driver.FindElement(By.Id("Temperature"));
wElement.SendKeys(countryO.capital);
IWebElement wElementButton = driver.FindElement(By.Id("getWeatherInfo"));
wElementButton.Click();
Assert.NotStrictEqual("-", wait.Until(drv => drv.FindElement(By.Id("Temperature")).GetAttribute("value")));

driver.Quit();
}

}

Our experience with Selenium was easy to implement and quick in performing the necessary tests. Our tests successfully passed, demonstrating that Orbital Bus can work with web applications and have no negative impact over console apps. Selenium not only let us confirm Orbital Bus was working as intended with a web application, but it will afford us with the flexibility and performance needed to grow these tests in the future as we expand our use cases.

 

  • Maria
  • Reyes Freaner

Getting Started with Orbital Bus

You’ve heard people talk about enterprise service buses and you think it’s time you learned out to use one. You read an awesome blog post about this new thing called Orbital Bus and you think it would be a good project to play around with. Where should you start? Let’s start here.

Understanding the Architecture

I’m sure you’ve checked out our project README, but just in case you need a refresher here’s a quick overview of how Orbital Bus works.
Everything starts with the Producer and the Consumer. The Producer produces calls into the system. These calls can be synchronous requests or asynchronous fire-and-forget messages. What’s important is that the Producer is what initiates the action. The Consumer consumes messages off the queue. Both the Producer and Consumer are external to Orbital Bus. They might be third-party services, COTS products, or custom code applications made by your developer(s). The Orbital Connector is a library the Producer uses to get messages into the system. We have a whole project dedicated to connectors. The Connector uses RabbitMQ to pass messages to the Dispatcher. The Dispatcher listens for incoming messages, finds services via Consul, and sends messages to the Receiver via it’s queue. Receiver’s do the heavy lifting. They load custom libraries, transform messages, and use adapters to send messages to the Consumer.
Here’s a diagram to give you an idea of the general flow of information:
An overview of the Orbital Bus flow.
An overview of the Orbital Bus flow.

Getting ready

For this little test, let’s put everything on your local machine. You’ll need to prepare by installing two third-party components: Consul and RabbitMQ. We use these for service discovery and message communication respectively. If you want some help you can check out our more detailed instructions. Since Orbital Bus is ready to communicate with any RESTful web service, we’re going to use JSONPlaceholder. Feel free to check it out and get a feel for the kind of messages you want to send.

Build a Producer

The Producer is the instigator of the pipeline. It calls out using the Orbital Connector and RabbitMQ to get the Dispatcher communicating with other nodes. Since our current Orbital Connector is written in .NET, you’ll want a .NET application that references it. We have a NuGet package to make it simple. We have four methods for sending with the connector: synchronously, asynchronously, synchronously that can be awaited, and a one-to-many route. We recommend starting with a synchronous call. All the producer needs is the service ID for the destination service (which you add to Consul below) and a JSON-serialized payload.
For more detailed instructions on making a Producer, check out our How-To Guide. It’s got a thorough process with code samples and everything!

Use Code Generation

Next we’ll setup the side of the Consumer. As we said above, we’re not going to bother building a web service (though you can if you really want to).
To get started you’re going to have to download the Code Generation project. We made this tool to help generate the necessary libraries for the Orbital Bus Receiver to connect with a service. All the files you work on for Code Generation are Javascript, so your C#, Java, Python, and Ruby developers should all be able to use it. Of course we have a handy guide to making a library. When you’re done building your library keep track of the `bin` folder in the project directory. We’re going to need all its contents.

Configure your Nodes

I know what you’re thinking: “Where’s the actual Orbital Bus?” That’s the beauty of our distributed system. The bus has no central hub to stand up. Each service has a node or nodes that live alongside it to facilitate communication.
To get our local instance up we’ll need both a Dispatcher and a Receiver node. You can download them on our release page. With the release package unzipped in a location of your choosing, you’ll want to copy over your code generation output. Remember that bin folder we told you to keep track of? Copy all its contents into the Plugins folder for the Receiver. The Receiver will pull in those libraries at runtime and then it’s ready to communicate to your web service.
You’ll also want to set the values of the configuration files to the appropriate values for your local deployment. We have a handy article about all the configuration properties. Be sure to open up any ports you’re planning on using for your Dispatcher and Receiver!

Run!

Now it’s time to set everything in motion! If your Consul and/or RabbitMQ aren’t already running start them up. Start up your Receiver and register it with Consul. (We also have a Consul Manager tool in the release package. Check out this article to see how you can use it to register your service.) Start up your Dispatcher and your Producer and start sending messages!
If you’ve run into any snags or want a more thorough description, check out our How-To Guide. It describes each step in detail so you can see how every part of the process should be configured.
What’s next? Try implementing a second call. Check out our other documentation, like our Handshake Diagram to better understand the paths of the messages. Maybe add another Receiver with another web service to give you an idea of multiple nodes on the network. Hopefully this test will be a small step along your long future with ESBs. Enjoy!

  • Joseph
  • Pound

Introducing the Orbital Bus

Today, we are proud to announce the public beta launch of the Orbital Bus open source project. The Orbital Bus is a distributed Enterprise Service Bus (ESB) intended to make it easier for developers to implement smart, reusable, loosely coupled services. We believe that a peer-to-peer mesh of lightweight integration nodes provide a much more robust and flexible ESB architecture than the traditional hub/spoke approach. Please check out our public repository and documentation.

I have been working in Service-Oriented Architecture (SOA) and with Enterprise Service Bus (ESB’s) for the majority of my career. I can’t even count how many debates I’ve been in on the value of implementing an ESB and how a hub/spoke architecture is more sustainable than point-to-point integrations. In fact, for a number of years, I took the architectural benefits of an ESB for granted.

By 2013, I was starting to see a pattern of large enterprise clients struggling to adopt SOA not for architectural or technology reasons, but for organizational and cultural reasons. I came to the realization that while ESB’s were being sold as bringing about a certain type of architectural rigor, it could only do that if the organization was very hierarchical and centralized. The ESB is really not well suited for an IT organization made up of more distributed teams and governance structures.

We thought of a better way to solve the real-time integration problem. With the help of some funding from NRC’s IRAP program, we started development of the Orbital Bus in 2015. The goal was to solve some of the major shortcomings that we see in traditional ESB’s:

Single Point of Failure – An ESB creates a single point of failure, as all common integrations must pass through it. Enterprises spend a significant amount of money to improve the availability of their ESB infrastructures (e.g., hardware redundancy, DR sites, clustering). What if the responsibility of translation and routing was pushed out to the edges to where the service providers are hosted? There would be no point of failure in the middle. The availability of each service would purely be dictated by the service provider and the lightweight integration node sitting in front of it, which means one node going down wouldn’t impact the rest of the ecosystem.

Implementation Timelines and Cost – ESB’s take a long time and a lot of money to stand up. A lot of effort is needed to design the architecture so it’s future proof and robust enough for the entire enterprise. And then there’s the cost of the infrastructure, software licenses, and redundancy. Never mind the cost of bringing in external expertise on whichever platform is being implemented. What if the platform actually promoted more organic growth? Each node is lightweight and could be stood up with no additional network zone or infrastructure. Developers will be able to build decoupled interfaces between handful of systems in a matter of days rather than months. And instead of needing to fiddle with complex platform configurations and go through 200+ page installation guides, the ESB could be stood up with a handful of scripts and created native service bindings in commodity languages such as C#.

Developer Empowerment – ESB’s move the responsibility of creating decoupled interfaces from the developer of the service into a central ESB team. It’s no surprise that nearly every SOA program I’ve ever worked on faced significant resistance from the development teams. And let’s face it, most IT organizations are poorly equipped to handle major culture changes, and that resistance often results in the killing of a SOA program. What if the architecture actually empowered developers to build better and more abstracted interfaces rather than try to wrestle control away from them? The platform would promote a contract-first implementation approach and generate all the boring binding and serialization code so developers can focus on the more fun stuff. By having the service interface code artifacts tied more closely to the service provider code, it opens up opportunities to better manage versions and dependencies through source control and CI.

We’ve had a lot of fun designing and developing this product over the last two and a half years. We are excited to offer the Orbital Bus to the community to collaborate and gather as much feedback as we can. Working with the open source community, we hope to create a more efficient and developer-centric way of integrating across distributed systems. We hope you will join us on this journey!

  • Shan
  • Gu