Dataweave Code to Identify Duplicate Key Values in JSON Array

This document covers Dataweave code to identify and flag duplicate field or key values in a JSON array. This code will treat the first copy as original and mark the remaining copies as duplicate. For this demonstration, we are using Mulesoft's Online Dataweave Playground - Link as an IDE.

Let’s begin...

Solution Preview

Code Snippet

%dw 2.0
output application/json
fun getDistinctRecords(records, field) = flatten((records groupBy ((item, index) -> item[field])) pluck ((value, key, index) -> value) map ((item, index) -> item distinctBy ((item, index) -> item[field])))
var distincValues = getDistinctRecords(payload, "Username")
var duplicates = payload -- distincValues
---
(distincValues) ++ (duplicates map ((item, index) -> item ++ {"Duplicate Flag": "Yes"}))

Brief Explanation

The main logic for this problem revolves around the distinctBy function. But you are here because you also found out about this function's limitation when dealing with JSON arrays. It is only good when we are trying to find duplicate JSON objects as a whole in an array. But, it doesn't work well when we have to identify duplicate JSON objects based on one field or key value.

In the code above, we have a user-defined function getDistinctRecords which takes a JSON array as input and the field or key name, which has to be checked for uniqueness. It outputs all distinct JSON objects based on the field or key name. We then separate out duplicates using the -- operator and the original payload. Lastly, we concatenate the two arrays and also add a Duplicate Flag key with a value Yes, just for fun. Hence, this code treats the first copy as original and mark the remaining copies as duplicate. Feel free to modify the code however you like as per your use-case.

Sample Input

[

  {

    "Username": "Alpha123",

    "Name": "Alpha"

  },

  {

    "Username": "Beta456",

    "Name": "Beta"

  },

  {

    "Username": "Charlie789",

    "Name": "Charlie"

  },

  {

    "Username": "Delta012",

    "Name": "Delta"

  },

  {

    "Username": "Echo345",

    "Name": "Echo"

  },

  {

    "Username": "Alpha123",

    "Name": "Alpha 2"

  },

  {

    "Username": "Delta012",

    "Name": "Delta 2"

  }

]

Output

[
  {
    "Username": "Alpha123",
    "Name": "Alpha"
  },
  {
    "Username": "Beta456",
    "Name": "Beta"
  },
  {
    "Username": "Charlie789",
    "Name": "Charlie"
  },
  {
    "Username": "Delta012",
    "Name": "Delta"
  },
  {
    "Username": "Echo345",
    "Name": "Echo"
  },
  {
    "Username": "Alpha123",
    "Name": "Alpha 2",
    "Duplicate Flag": "Yes"
  },
  {
    "Username": "Delta012",
    "Name": "Delta 2",
    "Duplicate Flag": "Yes"
  }
]