Database Enhancement - Merging Linked Records

Tools: Airtable, n8n, Make/Integromat

The Problem: Nearly 75,000 records that had duplicates and linked records across 3 tabs. I needed to merge the duplicates while keeping the linked records intact. 

Available solutions were able to merge duplicates but could not combine the linked records. 
Nor could I find any Airtable scripts that could dedupe and maintain the linked records. 

The Process

I start my process by identfying big issues that I don’t know the answers to. 

In this case, how to combine linked records and return them via API when it ranges from 1-2000+ linked records. 
To test this, I turned to the Airtable API docs to determine JSON structures and then built them out in Make. 

Lessons Learned

This was a fun solve in building out the right JSON structures for the Airtable batch API calls, optimizing iterators and aggregators to deal with the limits of Airtable (10 records max in a batch API call), and also figuring out how to deduplicate thousands of records. 

If I were to tackle this again, I would likely move everything into an SQL database and run SQL queries. This wasn’t a paid project so I was more interested in exploring the limitations of, n8n and Airtable. 

But all in all, very happy I took the time to solve this problem as doing batch API calls within Airtable has come up time and time again for clients.