Removing duplicates from an array of objects

I have a style of programming that generally works well for me. I like to solve a problem and then step back and evaluate my solution. This gives me a chance to see if there is a better way to solve the problem at hand. In the programming world we gave this a fancy name, refactoring. I am also very lucky to have some really smart people at work so when I have the chance to discuss code I take it.

Last week I was working on a problem that I have solved countless times before. I had an array of objects and I wanted to make sure no duplicates of a certain key were returned to the caller. Before I write some code I like to talk the solution out so in plain English this is what I need to do.

  • create an empty array to hold our new array
  • loop over our existing array of objects
  • if the current key doesn’t exist in our new array, add it

Seems pretty simple and I am betting a lot of people have taken this approach before. So what does this look in code? First we have our example array of objects. In this array class code is the key we don’t want to duplicate. Next we create an empty array that will contain our duplicate free result.

Next we are going to loop over our array. During each iteration we loop through our new array and check to see if that value exists, if it does not go ahead and add this object to the new array.

While this works great I was talking with a coworker (Jason Delmore) and he came up with a pretty cool alternative. Instead of creating a new array to hold our result we can simply look ahead to see if this key exists and if it does remove it and decrement the loop index.

We can also do the same thing in JavaScript.

This may seem like a small change or there may be better ways to solve this (I would love to hear them) but I love it. I enjoy looking at problems like this and understanding ways to solve them (in English) and then taking a look at how we can implement them.

Categories: ColdFusion,JavaScript

About The Author

My name is Dan Vega and I am a Software Developer based out of Cleveland OH. I love to play with new technologies and write about my experiences here. When I am not busy being a full time geek I love to lift heavy weights and hang out with friends and family. If you have any questions please don't hesitate to contact me.

Follow me on:
  • http://flexoop.com Gareth Arch

    I’ve taken to doing this kind of thing via struct. If the objects are exactly the same, and you can check some kind of id/key, they I usually just do something like this:

    local.objDedupe = {};
    for ( local.i = 1; local.i <= arrayLen(tmp); local.i++ ) {
    local.objDedupe[ tmp[ local.i ].myIdOrKeyHere ] = tmp[ local.i ];
    }

    then if you need to convert it back to an array, loop back over the collection and arrayAppend.

    This way you’re only looping over the array 1 (or 2) times to dedupe everything.

  • http://therealdanvega.com Dan Vega

    In my case though the objects were not always the same, just the key. If the objects are the same though couldn’t you just take a similar approach to what I am doing above using arrayFind()? Array find will search for simple or complex objects such as structures. I originally wanted to use this approach but quickly realized my objects were going to be different.

  • http://flexoop.com Gareth Arch

    You certainly could, but you would be going (potentially) through the loop quite a few more times, wouldn’t you?
    e.g. for 100 items
    The "struct" method:
    100 for loop over array, then 100 for struct back to array if there were no dupes, less if there were (as second loop would have the dupes removed)
    The "search ahead" loop:
    100 for first loop, 99 for next, 98 for next, etc. if no dupes were found. It’s certainly less than doing the standard "if not found loop over all items", but once you reach the 3rd item, you’ve performed more comparisons that the "struct" method.

  • http://therealdanvega.com Dan Vega

    I thought for some reason arrayFind let you specify a starting position. It will return to you the first index or 0 if not found. If you were able to supply an index you could do something similar by searching ahead but scratch that.

  • http://www.byteout.com intnick

    You could also do something similar to Gareth’s solution and use the struct only to hold your keys and check for duplicates there. Something like this (untested):
    keys = {};
    new = [];
    for ( i = 1; i <= arrayLen(tmp); i++ ) {
    if (not StructKeyExists(keys, tmp[i].myKey)) {
    ArrayAppend(new, tmp[i]);
    keys[tmp[i].myKey] = true;
    }
    }

    This will on the other hand consume a bit more memory because of the struct…

  • Arvind

    Great tip, can’t thank you enough for this.