Platform Engineering

Data Mapping with Groovy - Part 2

Posted by Admin on 02 December 2013

tech, mule, groovy

Part 1 of this series layed the foundation for some Groovy concepts and what makes the language suitable for data mapping tasks. With that in mind, lets dive into some of the advanced mapping features and some real world samples.

Introducing GroovyMap

There are 3 data transformation scenarios which we commonly encounter:

  • xml to xml – Reads an inbound xml and maps it to a different XML structure.
  • xml to map – Reads an inbound xml and maps it to a java.util.Map. Maps and Lists are very common data structures in Mule applications, being used for both JSON data formats and when using the Database transport.
  • map to xml – Reads an adapter response in the form of a java.util.Map, iterates over the enumeration of keys and maps that to a response xml.

We notice these transformations often involve a lot of plumbing code:

  • Accept an array of mapping inputs and an array of ‘helper’ objects (e.g. lookup tables, DB connections)
  • Initialise a Groovy builder object to help construct the mapping output. Perform the data mapping using the result Builder object.
  • ‘Serialise’ the result in a certain way (string, DOM tree, POJOs etc)

To rapidly churn out mapping code, it makes sense to move this boilerplate code into a common script which developers would import. This script provides a foundation involving the  “Builders” which are used across all our mapping code and the serialisation approach.

Here’s what a light weight might look like:

  package au.com.sixtree.esb.mapper
   
  import au.com.sixtree.esb.mapper.GroovyMap;
  import au.com.sixtree.esb.mapper.DOMBuilder;
  import groovy.json.JsonBuilder
  import groovy.xml.MarkupBuilder
  import groovy.xml.NamespaceBuilder;
  import groovy.xml.XmlUtil
   
   
  /**
  * Data Mapping utility that lets developers map between data structures and formats using Groovy Builder syntax.
  */
  class GroovyMap {
   
  private Object[] inputs
  private BuilderSupport builder
  private Object[] helpers
  private Closure mapping
  private Closure outputSerialiser
   
  private GroovyMap(Object[] inputs, BuilderSupport builder, Object[] helpers, Closure mapping, Closure outputSerialiser) {
  this.inputs = inputs
  this.builder = builder
  this.helpers = helpers
  this.mapping = mapping
  this.mapping.delegate = this
  this.outputSerialiser = outputSerialiser
  this.outputSerialiser.delegate = this
  }
   
  public Object map() {
  return outputSerialiser.call(mapping.call())
  }
   
  public static GroovyMap toXmlDom(Object[] inputs, Object[] helpers, Closure mapping) {
  def domBuilder = DOMBuilder.newInstance()
  return new GroovyMap(inputs, NamespaceBuilder.newInstance(domBuilder), helpers, mapping, { it.getOwnerDocument() })
  }
   
  public static GroovyMap toXmlDom(Object input, Closure mapping) {
  return toXmlDom([input] as Object[], null, mapping)
  }
   
  }
view rawGroovyMap.groovy hosted with ❤ by GitHub

You can see the full constructor for the GroovyMap is quite involved. To make life easier for developers, we provide static factory methods to create pre-built GroovyMaps for common scenarios (e.g. transforming to XML DOM). The final method is a convenience method for the simplest, most common mapping scenario: a single input object to a single output with no helpers.

Now let’s take a look at some examples of the common data mapping scenarios:

XML to XML

  //import the GroovyMap
  import au.com.sixtree.esb.mapper.GroovyMap
  //We have a customized version of DOMCategory helper class, this provides us with xpath like features to access elements
  //The default version involves a slightly different syntax.
  import au.com.sixtree.java.esb.mapper.DOMCategory
   
  //GroovyMap's toXMLDom() method takes input as the root/documentElement of the XML tree, type @Element
  //very similar to accessing the root element in Java - payload.getDocumentElement()
  return GroovyMap.toXmlDom(payload.documentElement) {
  //inputs is a list of input arguments
  def _in = inputs[0]
  //Groovy provides a neat feature called use(TYPE){} closure using which you can retrieve xml elements using xpath style syntax
  use(DOMCategory) {
  /* map the root element with a namespace
  The builder object is part of GroovyMap (of type NamespaceBuilder), which has a method called declareNamespace(namespace_map)
  */
  builder.declareNamespace('en':'http://sixtree.com.au/system/inboundadapter/v1/xsd' , 'xsi':'http://www.w3.org/2001/XMLSchema-instance')
  //Using the same builder object, construct the root element <en:getbookInventoryResponse>
  builder.'en:getbookInventoryResponse' {
  /*
  * The below construct is trying to represent a <book> with its Inventory Status across various locations
  * One book can be stocked at various locations
  * */
  book{
  //If condition to check if the input xml contains an element called <identifier>
  if(_in.book[0].identifier)
   
  //This creates an identifier tag <identifier>12345</identifier>
  //You can also use the element(value) style to achieve a similar result
  //eg. identifier(_in.book[0].identifier)
   
  identifier _in.book[0].identifier
   
  if(_in.book[0].name)
  name _in.book[0].name
   
  /*Since one book can have multiple inventory locations and details, iterate over the inbound xml's inventoryDetails element list
   
  Note the "inventoryDetails_tmp" identifier, its important you dont use same identifiers for input and output variables
  Here inventoryDetails is present in both input and output xml, so while mapping to an output xml, ensure you are using a
  different id for the variable (eg. inventoryDetails_tmp is used to iterate over the inbound inventoryDetails)
  You wont notice this ambiguity until runtime.
  */
   
  for(inventoryDetails_tmp in _in.book[0].inventoryDetails)
  inventoryDetails{
   
  status inventoryDetails_tmp.status
   
  location{
   
  city inventoryDetails_tmp.location.city
  state inventoryDetails_tmp.location.state
   
  }
   
  inventoryCount inventoryDetails_tmp.inventoryCount
   
  }
  }
  }
  }
  }.map()
  //Calling the map, serializes (or marshals) the string to an xml
view rawxml2xml.groovy hosted with ❤ by GitHub

Java Lists / Maps to XML

This is a common requirement when working with Mule’s JDBC transport (for example).

  //import GroovyMap to get a handle to the transformation method - toXmlDom()
  import au.com.sixtree.esb.mapper.GroovyMap
  //DOMCategory to get us xpath style operations eg.getNode, getAttributes while traversing through the xml
  import au.com.sixtree.java.esb.mapper.DOMCategory
  import groovy.xml.XmlUtil;
   
  //import static codeset cross-referencing helper class, see below for lookups
  import au.com.sixtree.java.esb.mapper.ValueCrossReferenceHelper;
   
  return GroovyMap.toXmlDom(payload) {
  /*inputs is the args array, where the 0th element is the payload (passed in the line above)*/
  def _in = inputs[0]
  //use(Type){} - Construct to access xml elements
  use(DOMCategory) {
  /* add namespace to the output xml
  * use builder object (@BuilderSupport) to create xml elements
  * builder object (type NamespaceBuilder) has the declareNamespace(Comma_Seperated_Map), use that as a setter for namespaces
  */
   
  builder.declareNamespace('xsi':'http://www.w3.org/2001/XMLSchema-instance' , 'ca':'http://sixtree.com.au/system/inboundadapter/v1/xsd')
   
  //create root element and create its closure
   
  builder.'ca:getBookInventoryResponse' {
  // nested mapping for book with identifier
  book {
   
  name(_in[0].NAME)
  identifier(_in[0].BOOK_ID)
   
  //repeating field. Iterate over the inbound Map's keys enumeration
  for(inventoryDetailsResponse in _in) {
   
  inventoryDetails {
  //xref lookup using a custom Java method
  status(au.com.sixtree.esb.common.crossreferenceclient.ValueCrossReferenceHelper.lookup("Book Inventory Status", "Outbound_System_Name", inventoryDetailsResponse.INVENTORY_STATUS, "Inbound_System_Name"))
   
  location {
   
  city(inventoryDetailsResponse.CITY)
  state(inventoryDetailsResponse.STATE)
   
  }
   
  inventoryCount(inventoryDetailsResponse.INVENTORY_COUNT)
   
  }
   
  }
  }
  }
  }
  }.map()//Serialize the entire closure to create an xml. Check the GroovyMap.map() method for more details
view rawmap2xml.groovy hosted with ❤ by GitHub

XML to Java Lists / Maps

  // no need to use builder here, just raw Groovy map syntax
   
  //get DOMCategory to get access to GPATH style operations like getNode getNodeAt etc.
  import au.com.sixtree.java.esb.mapper.DOMCategory;
   
  //declare a java.util.Map to hold results
  def result = [:]
  //get input root element
  def _in = payload.documentElement
  use(DOMCategory) {
  /*using DOMCategory get access to various xml elements, this is like calling
  *_in.bookId.text() is equivalent to responseDoc.getDocumentElement().getElementsByTagName("book").item(0).getFirstChild().getTextContent());
  *result.bookId is equivalent to result.put("bookId" , "123456") in Java
  */
   
  result.bookId = _in.bookId.text()
   
  //Using ?: Java operator to create an if else condition
  /*
  * Set results("inventoryStatus") value only if the inbound element has a non-empty value
  * One of the setter calls a custom XREF method (ValueCrossReferenceHelper.lookup(String args...)) to retrieve the corresponding system value
  * */
  (_in.inventoryStatus.text() != null && _in.inventoryStatus.text()!="")?(result.inventoryStatus = au.com.sixtree.esb.common.crossreferenceclient.ValueCrossReferenceHelper.lookup("Book Inventory Status", "Outbound_System_Name", _in.inventoryStatus.text() , "Inbound_System_Name")): (result.inventoryStatus = "");
   
  }
  return result
view rawxml2map.groovy hosted with ❤ by GitHub

In this case the GroovyMap helper class is not needed, because Groovy List/Map data structures are easy enough to create without a builder

Other helper classes

You’ll notice in the above code that we use our own versions of the standard ‘DOMBuilder’ and ‘DOMCategory’ classes. We made some small patches to the GDK classes to make our mapping code behave the way we wanted:

  • The patched DOMBuilder class constructs a full org.w3.dom Document object with the top-most element created by the builder inserted as the document element.
  • The patched DOMCategory class tries to convert each expression value to a string if it has a sensible string representation (i.e. it is not a list of elements). This is to replicate the handy behaviour of XPath and means we don’t have to add .text() or .toString()to the end of every mapping statement.

Working with Groovy in Eclipse/Mule Studio

Since Mule Studio is based on the Eclipse platform, you can install a Eclipse plugin to get excellent editor and debugging support for writing Groovy mappings without leaving your IDE.

To add Groovy support to your eclipse based IDE, install the GRECLIPSE plugin:

  1. Help → Install New Software → In the “Work With” text field add
  2. Add the Groovy editor support and Groovy 1.8.6 runtimes.
  3. Restart when requested.

Once this is installed, right click on the Project → Configure → Convert to Groovy Project. This should add the Groovy DSL Support Libraries to the classpath. It also adds Groovy Libraries to the classpath. However, if you’ve already added that in your pom, get rid of it using the “Configure Build Path…” link in eclipse.

 

If you like what you read, join our team as we seek to solve wicked problems within Complex Programs, Process Engineering, Integration, Cloud Platforms, DevOps & more!

 

GET IN TOUCH!

Leave a comment on this blog: