Author: Shan Gu

IT Organizations Need to Practice More, Dunk Less

Whenever I walk into a new client, the first things I hear from the Technology Executives are typically: “We need to modernize”, “We need to transform”, “We need to adopt <insert trendy tech buzzword>”. What I never hear is: “We need to bring our development and testing methodologies up to date”, “We need more collaboration across our teams”, “We need to inventory our skills and see what’s missing”.

If we think of the IT organization as a basketball team, that would be the equivalent of the coach saying: “We need more 3-pointers”, and “We need those fancy new shoes to be able to dunk”.  Whereas even the most inexperienced youth coach knows that the key to winning includes: “We need to practice dribbling and footwork”, “We need to communicate better on the court”, and “We need to improve our free throws/jump shots/rebounds”.

While it is both valid and necessary for IT organizations to push towards the big picture objectives highlighted by glossy Gartner and Forrester whitepapers, these have to be supported by continuous and deliberate investment in foundational concepts.

Let me step in as coach for a moment and propose a strategy for focusing on the foundation…

1)    Invest in the basics: Invest in good basic IT delivery concepts, kind of like dribbling, footwork, and basic fitness in basketball:

  • Make Business Analysis about teasing out the requirements from the Business’ objectives, rather than simply asking the Business to write down their requirements
  • Encourage good testing rigor and embed it throughout the entire solution delivery lifecycle, and not just at the end just before go-live
  • Promote good documentation habits and create templates for common documents (e.g., logical solution architecture, functional designs, interface specifications, data models)
  • Spend adequate time and budget to implement solutions which improve developer productivity (e.g., continuous integration, 3rd party frameworks)
  • Allocate budget for developers to learn different languages so they can be exposed to different software concepts and improve their coding skills
  • Spend generously on training for system analysis, modeling, design methodologies (e.g., domain driven design, SOA, microservices architecture, semantic modeling, BPMN), and not only on those being standardized by the organization, but to improve people’s ability to make smart decisions

2)    Communication is key: Create an environment that promotes collaboration and teamwork:

  • Create communities of practice across your organization (or connect to external groups) to build on collective knowledge and experience
  • Implement real-time collaboration tools (no, Sharepoint and instant messenger don’t count)
  • Make governance less about formal approvals and more about ensuring the right expertise is pulled in at the right stage of a given project
  • Adopt iterative delivery methods to promote frequent touch points between IT and Business obtaining feedback and ensuring alignment

3)    Focus on the right skills: Build the skills that support your strategic objectives. After all, dunking is only made possible by training to jump higher:

  • Strengthen Information and Data Management capabilities as a foundation for Big Data Analytics
  • Educate the team on hashing algorithms, binary trees, digital contracts, and distributed storage to bring Blockchain to the table naturally
  • Leveraging Cloud means good distributed system design, loosely coupled interfaces, container-ready applications, and security frameworks that can deal with 3rd party infrastructure
  • Adopting COTS requires strong technical Business Analysis, ability to negotiate requirements with the Business, and strong platform administration skills

We all want to work with the cool new tech and follow the latest trends. Working with the latest and greatest is what draws people to technology work. But the team will be stronger if the foundation is strong and the team is well connected so take time to build our own skills and our teams’ foundations so we can all up our game.

  • Shan
  • Gu

Introducing the Orbital Bus

Today, we are proud to announce the public beta launch of the Orbital Bus open source project. The Orbital Bus is a distributed Enterprise Service Bus (ESB) intended to make it easier for developers to implement smart, reusable, loosely coupled services. We believe that a peer-to-peer mesh of lightweight integration nodes provide a much more robust and flexible ESB architecture than the traditional hub/spoke approach. Please check out our public repository and documentation.

I have been working in Service-Oriented Architecture (SOA) and with Enterprise Service Bus (ESB’s) for the majority of my career. I can’t even count how many debates I’ve been in on the value of implementing an ESB and how a hub/spoke architecture is more sustainable than point-to-point integrations. In fact, for a number of years, I took the architectural benefits of an ESB for granted.

By 2013, I was starting to see a pattern of large enterprise clients struggling to adopt SOA not for architectural or technology reasons, but for organizational and cultural reasons. I came to the realization that while ESB’s were being sold as bringing about a certain type of architectural rigor, it could only do that if the organization was very hierarchical and centralized. The ESB is really not well suited for an IT organization made up of more distributed teams and governance structures.

We thought of a better way to solve the real-time integration problem. With the help of some funding from NRC’s IRAP program, we started development of the Orbital Bus in 2015. The goal was to solve some of the major shortcomings that we see in traditional ESB’s:

Single Point of Failure – An ESB creates a single point of failure, as all common integrations must pass through it. Enterprises spend a significant amount of money to improve the availability of their ESB infrastructures (e.g., hardware redundancy, DR sites, clustering). What if the responsibility of translation and routing was pushed out to the edges to where the service providers are hosted? There would be no point of failure in the middle. The availability of each service would purely be dictated by the service provider and the lightweight integration node sitting in front of it, which means one node going down wouldn’t impact the rest of the ecosystem.

Implementation Timelines and Cost – ESB’s take a long time and a lot of money to stand up. A lot of effort is needed to design the architecture so it’s future proof and robust enough for the entire enterprise. And then there’s the cost of the infrastructure, software licenses, and redundancy. Never mind the cost of bringing in external expertise on whichever platform is being implemented. What if the platform actually promoted more organic growth? Each node is lightweight and could be stood up with no additional network zone or infrastructure. Developers will be able to build decoupled interfaces between handful of systems in a matter of days rather than months. And instead of needing to fiddle with complex platform configurations and go through 200+ page installation guides, the ESB could be stood up with a handful of scripts and created native service bindings in commodity languages such as C#.

Developer Empowerment – ESB’s move the responsibility of creating decoupled interfaces from the developer of the service into a central ESB team. It’s no surprise that nearly every SOA program I’ve ever worked on faced significant resistance from the development teams. And let’s face it, most IT organizations are poorly equipped to handle major culture changes, and that resistance often results in the killing of a SOA program. What if the architecture actually empowered developers to build better and more abstracted interfaces rather than try to wrestle control away from them? The platform would promote a contract-first implementation approach and generate all the boring binding and serialization code so developers can focus on the more fun stuff. By having the service interface code artifacts tied more closely to the service provider code, it opens up opportunities to better manage versions and dependencies through source control and CI.

We’ve had a lot of fun designing and developing this product over the last two and a half years. We are excited to offer the Orbital Bus to the community to collaborate and gather as much feedback as we can. Working with the open source community, we hope to create a more efficient and developer-centric way of integrating across distributed systems. We hope you will join us on this journey!

  • Shan
  • Gu

Embracing the Technology Meltingpot

One of the most common objections I hear among my large enterprise and government clients when discussing adopting new technologies is “We’re a Microsoft shop so why would we look at a Java-based tool?” or open source, or Google, or Salesforce, and the list goes on.  This objection is grounded in the opinion that increasing the technology mix increases complexity, and thus increases the operational risk and cost.

However, the biggest challenge for IT executives has shifted from tightening their operational budgets to managing the constant risk of technologies becoming unsupported or having a vendor development path that no longer aligns with the enterprise’s needs.  The technology market is evolving and changing faster than ever.  Programming languages grow and wane in popularity and support over a cycle of just a couple of years; new frameworks breath new life into technologies that were previously left to die; acquisitions can make entire enterprise platforms obsolete overnight; and new innovations are happening constantly throughout the IT stack from networking, to virtualization, to application platforms, to business applications.

In such a rapidly changing and unpredictable environment, the best approach to managing risk (as any good investment adviser will tell you) wordleis to diversify.

In fact, any IT organization that doesn’t have an openness to innovate and investigate new technologies will ultimate die a slow death through obsolescence and operational bloat.

Instead of being afraid of the operational complexity and cost introduced by shaking up the technology stack, IT executives should be embracing them as opportunities for their teams to develop new skills and to gain a wider perspective on how to implement solutions.  Instead of remaining within the comfortable confines and protections of a given development framework, developers should be pushed to understand how different technologies interoperate and the importance of having disciplined SDLC methodologies to deal with complex deployments.

The key to success in all of this is integration.  Developing mature integration practices like modular design, loose coupling, and standards-based interoperability ensures that new technologies can be plugged into and unplugged from the enterprise without cascading impacts on existing systems.  Disciplined SDLC methodology especially around configuration management and change control allow different technology teams to work in parallel, resulting in more efficient project delivery.

IT organizations must adopt a culture of openness and curiosity to from the inevitable changes to their technology ecosystem.  They must invest in mature and disciplined integration practices to effectively manage those changes.

  • Shan
  • Gu

Prescriptive Governance Leads to Shadow IT

Let’s face it, to most people in IT, “Governance” is a dirty word.  That perception is not born out of an idea that IT governance is bad, but out of the reality that IT governance is badly implemented in most organizations.  When an organization confuses good IT governance with overly detailed and prescriptive IT governance, it starts to constrain rather than enable its people.  And when people feel constrained and not empowered to make the decisions, they work around or against the process, which then results in proliferation of shadow IT.Governance_ShadowIT

The reason for this phenomenon is that many organizations approach IT governance with a few very flawed assumptions around how software and technology projects work:

  1. Changes are bad;
  2. Consistent results are driven from consistent processes;
  3. Standardization reduces the number of decisions, which makes the process more efficient and consistent; and
  4. Measure of a good project is on-budget and on-time.
These assumptions are fatal to any IT organization because they fail to recognize the realities of the nature of technology and the people implementing it:
  1. Technology is about change.  The whole point of implementing technology is to support and enable more rapidly changing business needs.  Add that to the speed of technology changes, the default position of any good IT governance process should be to assume constant change and deal with it head on instead of trying to contain and avoid it.
  2. Speaking of change, there is no way for a one-size-fits-all process to anticipate all the different ways a technology project can unfold.  In fact, the more prescriptive a process is, the less likely it will fit the next project.  There are simply too many variables and moving targets in modern enterprise IT projects.
  3. You hired smart people to implement technology and guess what?  Smart people like to make decisions and feel ownership of their work.  By over-standardizing, talented tech resources are turned into the IT equivalent of assembly line workers.  At best they become disengaged and stale in their skills.  But more likely, they take matters into their own hands and create opportunities to make decisions or fight the governance process to retain some ownership of the work they’re being asked to do.
  4. IT initiatives exist to make your business better and the users happier.  While budget, scope, and schedule are important, they’re management measures on the process rather than whether a project was truly successful.
So how do we fix this?  In a word, simplify!  And here are some things to think about when slimming down your IT governance process:
  1. Reduce and align the number of gates to the number of “point-of-no-return” decisions on a project (e.g., business case, functional design, technical design, go-live).
  2. For each gate, focus on what kinds of decisions need to be made, guidance on people who should be involved, and some basic examples of information that should be provided as input.  Let the smart people do what they’re being paid to do, which is talk it out and own the decision.
  3. Standardize only the mundane and highly repeatable decisions.  Standards are about helping speed up the process and focusing the effort of only debating things that deserve to be debated.  It’s not about compliance metrics and enforcement.  If you have to put in an exception or exemption process, you’ve gone too far.
  4. Ensure communications on what the project will deliver in terms of functionality and value.  Most stakeholders care a lot more about whether a particular feature set is being implemented for their users rather than whether a particular deliverable is green or yellow.
In the end, this is about creating a process that helps to focus people on the real objectives of the business and fostering communications.  It’s about assuming that people are intelligent and reasonable and capable of making good decisions when given the opportunity.  And if that turns out not to be the case, it’s a HR problem and not something that should be fixed with yet more governance processes.
  • Shan
  • Gu

The Quiet Evolution of SOA

We’ve all found ourselves looking at an organization’s web services and commenting on how “It’s not really SOA”. Maybe because the program still maintains point-to-point interfaces, or maybe the organization hasn’t put in place any form of governance, but for whatever reason, we declare that it simply isn’t comprehensive enough to be considered SOA. That begs the question then: who is actually doing TRUE enterprise wide SOA? Well… very few organizations. Anne Manes famously declared that “SOA is dead” back in 2009. So why is it that we still find ourselves evangelizing and building towards this vision?

The answer is that our understanding of what makes an SOA program successful has quietly evolved over the last few years. Enterprise-wide re-platforming and re-architecture initiatives gave way to tactical adoption of SOA. The success of SaaS and BPM adoption meant that organizations are implementing the principles of service orientation without explicitly calling it an SOA program. And instead of trying to figure out just how to effectively measure SOA ROI at the enterprise level, much greater success has been found measuring the value created within a given portfolio and/or capability.

So while we Architects have not given up on the hope of achieving SOA utopia, we have become more realistic in our approach:

  1. Identify a very specific problem to solve with an SOA approach, be it to reduce the time-to-market of a frequently changing business process, or to reduce the application footprint of a given line-of-business.
  2. Demonstrate the value of SOA by successfully solving that problem.
  3. Rinse and repeat.

At the end of the day, any plans for enterprise level SOA can only be built a critical mass of successful self-sustaining SOA capabilities/portfolios.

  • Shan
  • Gu

Fractal Governance

SOA landscapes today look very much like fractals. An organization may have several internal capabilities presented as reusable services that connect to each other. It may even connect to 3rd party and/or cloud based services. But if you drill down into each of these services, you’ll likely see a composite application that is made up of several finer grained services interconnected together. And as a math geek, I am naturally curious about all things related to fractals.

In fact the fractal pattern appears in almost anything that’s responsible for connecting things together: highway and road systems, power grids, the internet… the list goes on. In all of these systems, there exists a hierarchical system of management and governance to regulate its functionality. Each country, for example, have national standards and regulatory bodies that define how power is to be exchanged, managed, and consumed. At the regional levels, there are additional standards and regulatory bodies that deal with region-specific decisions such as how much power to generate, pricing, and what equipment is to be installed where. Similar structures are true for transportation and telecommunications. So why is it that most organizations see SOA governance as an all-encompassing enterprise wide responsibility?

The interaction requirements and lifecycle characteristics of enterprise level composite services or business processes are very different from those of a utility service. To paint the entire enterprise service landscape with an uniform set of standards and processes will either result in a high number of exceptions or a lowest common denominator scenario. To be effective, an organization’s SOA governance model must match its SOA deployment model. The governance model must exist not just at the top, but at a granularity that matches how the services are being deployed and managed. Service Portfolio Managers, then are not just another role within the governance model, but micro versions of governance domains themselves. Service Portfolio Managers must be allowed to define their own standards and processes that are appropriate for the specific services that they’re responsible for. The SOA governance model for the enterprise must consider what standards and processes are appropriate for all services, which are appropriate for only the ones being consumed across the enterprise, and which ones should be left up to the Portfolios to govern themselves.

  • Shan
  • Gu

Is Service Oriented Architecture (SOA) Still a Thing?

The Oldest Buzzword Around

Service Oriented Architecture (SOA) isn’t a new concept by any means.

It’s practically a decade old and, in IT years, that’s beyond the useful lifespan of just about all buzzwords. And that’s the problem; as a buzzword, SOA never attained the same level of popularity as Cloud or Big Data. The concept of SOA was nebulous and how an organization could achieve SOA was even more unclear.

Vendors were pitching anything ranging from just an asynchronous messaging infrastructure to a full blown process automation and orchestration suite as the “Conerstone of Enterprise SOA” solutions. Further confusion was caused by product vendors trying to differentiate their products by pushing the importance of interoperability standards (ie. WS-*) claiming that other competing products weren’t truly “SOA” for one reason or another.

This confusion created just as much negative stigma around the term SOA as positive sentiment. While product marketing folks were focusing on the discussion of just what is and isn’t SOA, the Architects were quietly picking and choosing the concepts of SOA that they liked and evolving their enterprises’ IT landscapes.

SOA – Still Alive & Kickin’

Fast forward to today.

RESTful has fully taken over as the web service integration style of choice for the Internet, relegating SOAP for internal enterprise interactions and transactions that are considered “low throughput”. JSON has gained traction in the same way over XML thanks to movement towards mobile computing and a renewed focus on making interfaces as lean byte-wise as possible. No one thinks twice about decoupling the UI from the business logic and integrating using a set of web service calls. And asynchronous messaging is practically the status quo method of propagating large amounts of data across the enterprise.

So yes, the key SOA concepts of:

  1. Developing applications that promote reuse
  2. Decoupling functional application components to improve flexibility and agility
  3. Standardizing the way interfaces are described and interacted with to promote predictable and consistent integrations are more prominent than ever. Exposing Big Data stores as RESTful services is one of the most popular ways of integrating with these technologies. And the SOA concepts of abstraction, service contracts, and reuse are at the foundation of SaaS solutions.

It Just Makes Sense

SOA is at a level of maturity where it no longer benefits from having its own buzzword.

After all, you don’t see organizations advertising that they’re a client-server shop or that they are prolific adopters of web architecture to differentiate themselves in 2013. We’re at a point where sound architecture principles put forward by the proponents of SOA nearly a decade ago, have become just good architecture practice.

The conversations today with IT executives should no longer be “Should you adopt SOA?” but “What should you do to better address reuse, flexibility, and consistency within your enterprise?”

  • Shan
  • Gu

Enterprise Service Bus (ESB): To Adapt(er) or Not to Adapt(er)

One of the most commonly touted features of commercial Enterprise Service Bus products (ESB’s) is the out-of-box adapters for other COTS products (eg. SAP, PeopleSoft, Siebel). But organizations who become dependent on these adapters and take a “use ‘em if you’ve got ‘em” approach inevitably find that the implementations are a lot more complex than advertised. This is because the decision on whether or not to use adapters should be driven by the alignment of the organization’s skillsets and support structure, not the perceived simplicity in using out-of-box components.

I’ve created a simple decision tree to help clients determine whether they’re better off using the adapters provided by their ESB vendor or whether they should consider some alternative method of integrating with the ESB:

  1. Is the integrating system a data component with no business logic layer and with no built-in ability to be exposed as a web service (eg. Database, LDAP, MQueue)?

    Yes – Use the provided adapters. There’s no point adding an additional layer of complexity when simple data integration is very mature among all the leading ESB products. Also ask yourself if you’re doing data mappings and translations within the adapters? Whether there will be a DBA on the ESB support team or not. ESB’s are excellent at handling simple data translations (ie. Field-to-field). Any complex data translations such as multi-table selects and data merges should be handled in the data tier where you have DBA teams who are equipped to manage them.

    No – Go to question 2.

  2. Does the integrating system have the built-in ability to generate a web service or ESB-compatible API?

    Yes – Go to question 3.

    No – Use the provided adapters. The whole point of SOA is to leverage existing assets as much as possible and to avoid building new custom code for integration. Putting in a custom coded service layer between the ESB and the target defeats the objective.

  3. Will the support team for the integrating system be cross-trained on configuring the ESB adapters?

    Yes – Use the provided adapters. The support team of the integrating system will ensure that the adapters and reconfigured as necessary based on any changes in the application in question.

    No – Expose web services and/or API’s on the integrating system. The support team for that system are the experts on that application. They know the data model and the implications of seemingly inconsequential details on the business logic in the backend. I’ve seen many an adapter implementation go off the rails because the team implementing the ESB simply are not experts on every system they’re connecting to. Let the team with the expertise manage the abstraction and register their endpoint on the ESB so you can still take advantage of features like version management and BAM.

Next time you’re opening the box to your shiny new ESB, resist the urge to take all the adapters out and deploy them everywhere possible. Ask yourself in the long run, who’s going to own the integration point? And whether their lives will be made easier with the adapter or without? An IT landscape that is well supported is much more important than an IT landscape that matches exactly to the vendor’s product sheet.

  • Shan
  • Gu

Agile Governance in SOA

I know, at first glance it looks like the cramming together of 3 loosely defined and oft-debated buzzwords into the title, but got your attention didn’t it? So if you will bear with me for a few minutes, I will explain how applying Agile to a much more abstract of an activity such as building an SOA Governance Framework works.

SOA Governance is a fluid concept, with many differing, complementary, and overlapping point of views. Is it a framework focused on lifecycle management of service assets? How does it integrate with existing governance frameworks around other IT components such as infrastructure, database, packaged applications? Or should it be focused around the monitoring and management of business capabilities. The discussion could go on forever.

Because of this fluidity, I’ve seen two major problems come up again and again with clients when it comes to creating an SOA Governance Framework:

  1. The client has invested in a shiny new SOA Governance Framework complete with templates, registries and repositories, architectural review boards, the works. It takes a year to roll out. And the moment it’s done, people start finding scenarios that don’t fit within the framework; do not apply it consistently; or refuse to use it altogether.
  2. Various architects within the client organization can’t agree on the scope, focus, or business case around such an ambiguous concept as SOA Governance, so it keeps on getting pushed off.

The reason why the first problem occurs is that organizations are treating Governance like a traditional application build project. You need to define it all up front and build it once. And that can only lead to an SOA Governance Framework, or any Governance Framework for that matter, which is out-of-date before it’s even delivered. Governance needs to constantly evolve to adapt to the changes in an organization. Is the organization moving from being IT focused to being business focused? Is there an increased uptake in COTS implementations over custom development? Is the organization moving to an offshore support model? All of these changes in the organization will impact how the services should be governed. In short, the way the SOA Governance Framework is developed and managed over time needs to be flexible and adaptable to change. Sounds like Agile, doesn’t it?

The same general mentality also leads to the second problem: the idea that the SOA Governance Framework has to be perfect and all-encompassing the first time around. The fact of the matter is that instead of being stuck in analysis paralysis, we’re always better off with having something rather than nothing. Even an imperfectly managed SOA landscape is better than a completely unmanaged landscape. One of the key tenets of SOA is to enable continuous improvement of business capabilities and efficiency by building it out in loosely coupled modules. So why can’t we implement a SOA Governance Framework by starting small and building up additional components as we learn more about the landscape and become more experienced in working with it. Isn’t it better to solve a real problem that we encounter than to create a solution in anticipation of a problem? Once again, sounds a bit like Agile, doesn’t it?

That was a long preamble. So here’s our approach:

  1. Get the organization to agree on a core set of SOA Governance deliverables, creating a baseline of the framework. In Agile terms, this list is our backlog and includes:
    – a Service Definition template
    – a Service Contract template
    – a Taxonomy
    – a Service Registry and Repository
    – and some kind of org chart from at least a support perspective
  2. For each deliverable defined in the backlog, analyze whether something similar already exists in the organization today or needs to be created. Group similar/relevant deliverables together (ie. Service Definition template with Service Contract template). Implement it and deliver it to the organization. These are our sprints.
  3. While operating our SOA landscape, constantly be on the lookout for areas to improve. Did we have any services that didn’t fit the definition template? Did we have a project that identified inefficiencies within the architectural review process? Is the service inventory becoming too cumbersome to manage? Are people adhering to the change management process and if not, why? Was there a re-org that requires us to rethink the key roles and responsibilities? Frame these into deliverables and put them into the backlog.

The key to success here is continuously looking at the applicability and appropriateness of the framework and keep it aligned to the rest of the organization. If the users of the framework start to feel like they have to shoehorn something into a structure that no longer works for them, they will cease to use it. So don’t fall into that trap and let’s apply some agility to our SOA Governance initiatives.

  • Shan
  • Gu