Integration Project Pre-Development Documentation

04 March 2017

Introduction

This article describes how to use the documentation templates for Visio 2016.

One of the most challenging part in BizTalk projects is the documentation, resp. specification, of the architecture and the design. The fundamental importance of documentation cannot be overstated.
I was using the UML notation for documentation of .NET applications and tried to use it for integration projects as well. Since there is too much information in the UML stencils not needed for a quick overview and basis for discussions I decided to design some easy to use shapes.

A BizTalk project can be very complex because in a busy IT environment, a lot can change in a short period. So, I need some easy to read and easy to maintain overview to give detailed enough information to start developing. Before I start programming I want to get the vision of the customer and what systems, entities, customers and vendors are involved in the processes for future automation.

There is a rule of thumb I follow when discussing the prerequisites for development: Tell me "What" you want, don't tell me "How" you want it. Talk about the process requirements from a business point of view. Nowadays, there is for almost every technical problem a solution, so don't worry about development details. Prosperous enterprises are successful because of their ideas and agility, not by the amount of technical stuff they use.

Here I share my templates and give you some information how to use them. This documentation is useful for developing Business Process Automation with BizTalk server, if you use this powerful integration platform just for data transformation and message routing you can skip this.

Prerequisites

Infrastructure

Purpose: Overview of Infrastructure

  1. Open template "infrastructure.vsdx"
  2. Open stencil "infrastucture_stencil.vssx"
  3. Activate "Shape Data Window"

This is the first template I use in my documentation. Assembling all information about the BizTalk environment. Customers should be aware of the need for three stages involved in the implementation:
Development – Test – Production.
If an information is missing at the time, I skip it and assemble the missing data during the development process.

For the development stage I use at least a virtual machine with BizTalk Developer Edition, SQL Server Developer Edition and Visual Studio on the same machine.
The test environment results from the existing infrastructure / needs on reliability and maintenance. My financial customers implement the whole production system for testing and the business departments must commit the testing results before deployment into production. They want to have a separate SQL Server to proof the performance results. Some customers do not want to have an explicit test system. From my point of view this is inacceptable, but: "The Customer is the Boss".
In production environment the BizTalk Server(s) and SQL Server(Cluster) are always on separate machines.

Next all the entities, for all processes, will be assembled. An entity could be a system at my customers site or an application/system on the customers' vendor/customer site. The purpose is to collect all the data for the upcoming bindings. When starting the project, there is normally a lack of information about the ports and bindings for the implementation. This is subject to change during development. For the ports, I use different colors. If the address is pointing to an external system, the information has red background to see immediately if a change of address/application/system has an impact on the implementation.

Drag and drop the Shapes to the sheet, if there is data associated you will get a popup for entering the information.
Click on the shapes to change/edit the data associated. Tip: Try to click on the text to get the data displayed. It's a little fizzy to get the right shape, play around with it and you will gain experience by using the template
All "…Server" shapes and the "Binding" Shape are data enabled. The rectangle is just for grouping the shapes, the connectors to display there have to be data exchanged with the BizTalk Server.


Flow Templates

For my BizTalk projects I always follow my own paradigm that I been working with since I started to implement solutions for integration projects. There is always an ESB implementation of a process from the customer's point of view. No messing around with applications or systems to integrate, try finding the pure description of the process with business terminology. Customers tend to use the terminology of their leading application where the process starts. But I want a flexible, agile design with the ability to react very quick if applications or systems change. For later support it's also very difficult to find the source of problems if you tell someone from the department: The Field NTANF in Segment E1EDT13 from Message DELVRY03 is no valid date. The other way: We received an Order Message in the Logistics Order Process with an invalid Deliverydate.

Processflow

Purpose: Determine Processes

This template is used to give an abstract view on "How" the process will be implemented in a BizTalk Solution. Per Visio sheet here, a deployment package for the BizTalk environment will result.

  1. Open template "processflow.vsdx"
  2. Open stencil "flow_stencil.vssx"

Each process identified for automaton gets split into three layers.

  • ESB
  • Entities delivering data to ESB
  • Entities consuming data from ESB

From the swimlane shapes drag and drop a horizontal swimlane to the sheet. Now give the process a name. I use abbreviations here, because in Visual Studio there is a limitation for the max length of a path and I want to use the documentation also as requirement specification for development. There is always a main process and a sub process in my projects e.g. Procurement, Accounting, Sales, Logistics… as main process, Payroll, Asset, Masterdata… as sub process of accounting

Now in the ESB there will be an implementation for the customer in BizTalk and in the vertical swimlane this will be documented. The naming convention is: "company.bt.lo.md.esb". If the customer's name is sunato münchen, the main process is logistics and the sub process is masterdata, it results in sunmuc.bt.lo.md.esb. Each name will result in a visual studio project name when development starts.

Next for each entity a vertical swimlane is placed left (delivering data) or right (consuming data) from the ESB. Same naming convention, but if the entity is a partner outside the customers scope the company name will change e.g. there will be logistics data delivered by the Contoso Corp. and their application is Dynamics AX it results in contoso.bt.lo.md.dax. Ditto for consuming partners or applications.

Give the kind of data routed in this process a name e.g. Customer. This means a data structure defined as a customer will be processed. It likely will happen that data will be transformed in other structures within the process, so put another horizontal swimlane to the sheet and name it as whatever structure it will be called e.g. Contact. In this sample a Customer is delivered and will be transformed to a Contact in different entities. Do not put to many details in the documentation at this stage. If we have to query another system for getting additional data to enrich the resulting structure, it will be documented in the message flow.

Each layer gets an orchestration shape. In the implementation this is not only an orchestration, there will also be schemas and maps. But we only want an abstract view without too many details. Give the shapes a name in most cases it will be the same name as the name of the data structure. Additionally, I put the version number and for entities the direction of the data flow. Versioning is an extremely important part in BizTalk Server. Many developers do not version their artefacts and later there is no chance to do a zero downtime deployment for changes in the process!

Last but not least put the IO shapes between the vertical layers. As you can see in the sample below the ESB does not have a physical connection to the outside world. I like the paradigm of a loosely coupled system, so only the entities will have physical ports. For the ESB all messages will be retrieved and submitted from/to the messagebox. On the entities side there will be communication with a physical port delivering the data from partners and applications. There should be a protocol for communication defined. If the port will be on my customers site it gets the local shape, if outside it gets the external shape. Later on when the solution is in production environment and support cases will show up, the first and second level support can easily identify communication problems on the partners site or on the local environment.


Messageflow

Purpose: Determine schemas and transformations

The message flow gives detailed information about messages that trigger a process and how the messages will be routed through all artefacts inside the BizTalk Server application to their destination.

  1. Open template "messageflow.vsdx"
  2. Open stencil "flow_stencil.vssx"

The Layout is the same as defined in the Process Flow, but with more detailed information about the data processed. The sample below defines a master data process where Dynamics AX sends a message with an id of a customer to signal a change or an inserted data record in the system. The message is delivered to the ESB, transformed to a customer message that query Dynamics AX to get the whole record with all data of the customer. Why don't query the whole record inside the dax entity? This depends on the needs of your implementation, maybe another entity just needs a signal a change in the record and triggers a different process. The customer is processed in the ESB and will be delivered to a web application (company.bt.lo.md.wap) and a legacy application (company.bt.lo.md.lap) as vendor or contact

Here you can see another dependency used for developing the solution: All entity projects reference the ESB messages (schemas) but the ESB projects never reference any of the entity messages. A change of an entity can now be easily implemented without impact to any of the other entities. But be careful! The ESB definition should be solid as a rock when referenced by the entities. A change here has impact on all entities subscribed to the ESB process.

With the development of the solution I change the name of the messages to real schema names (root node name). I do not include request/response patterns for pure technical purpose, if there is relevance from a business perspective the pattern will be documented e.g. Response for exception handling is not documented, response with data structures for the business process is included.

For the later support of the system I give the documentation to first and second level support to locate a reported problem for third level support.