API LIBRARY
Collapse Sidebar Menu
Expand Drawer

Replication Endpoint

For purposes of resource management, a standard query on Trestle produces a maximum of 1 million rows, which you can page through 1000 at a time. When you ask for a row over 1M, Trestle will return an error containing the message "Error getting data!".

However, there are valid queries you might run against big data sets that could legitimately contain over 1 million rows. To support the download of such big data sets, we've created the Replication Endpoint. This feature allows for fast and efficient querying of very large amounts of data.

Add "replication=true" to any query, and the first page of results will also provide a special URL that will give you the next page of results each time you hit it. Continue hitting it until its results are empty, and you will have downloaded the entire result set.

Note:
It is recommended to use a single thread for replication. If multiple threads are desired, you may split the query by zip code or ModificationTimestamp by the month etc. and run each replication on a separate thread. 

The unique replication URL that you receive in your first hit will expire after five minutes of inactivity, so be sure to request your next page of data shortly after the preceding query completes.

------------------------------------

For WebApi: 

STEP 1:  
Add new 'replication' query parameter to your regular query:
* &replication=true 
* https://api-prod.corelogic.com/trestle/odata/Property?$top=1000&$select=*&$orderby=ListingKey&replication=true 
STEP 2 (repeat until you get zero results):
In the result from STEP-1, you will get a "@odata.nextLink" node. Use that for getting the next set of results for each subsequent call.
For example: @odata.nextLink with below format should be used repeatedly for all subsequent calls.

* https://api-prod.corelogic.com/trestle/odata/Property/replication?id=123-a119257f-5dc0-4a47-bbb3-e526efd22656

------------------------------------

For RETS:

STEP 1:  
Add new 'Replication' query parameter to your regular query:
* &Replication=true 
* https://api-prod.corelogic.com/trestle/RETS/Search?SearchType=Property&Class=Property&Query=(ListPrice = 1000000)&QueryType=DMQL2&Format=COMPACT-DECODED&Limit=1000&StandardNames=0&Select=ListingKey&Orderby=ListingKey&Count=1&Replication=true


STEP 2 (repeat until you get zero results):

In the result from STEP-1, you will get a "NEXTLINK" node. Use that for getting the next set of results for each subsequent call.
For example: RETS NEXTLINK with below format should be used repeatedly for all subsequent calls.

* https://api-prod.corelogic.com/trestle/RETS/Replication?id=123-a119257f-5dc0-4a47-bbb3-e526efd22656

------------------------------------

Psuedo Code:

Build initial Query with filters and other settings. Append '&Replication=true' to the query.
From the response, 'nextLink' = value of "@odata.nextLink" [OData] or "NEXTLINK" [RETS]
while(recordCount != 0)
{
    Query Trestle with 'nextLink'    
    Process results
}
Done.



Examples

Code Example

https://api-prod.corelogic.com/trestle/odata/Property?$top=1000&$select=*&$orderby=ListingKey&replication=true

Result

{
    "@odata.context": "https://api-prod.corelogic.com/trestle/odata/$metadata#Property/CoreLogic.DataStandard.RESO.DD.Property",
    "@odata.nextLink": "https://api-prod.corelogic.com/trestle/odata/Property/replication?id=123-a119257f-5dc0-4a47-bbb3-e526efd22656",
    "value": [
        {
            "AboveGradeFinishedArea": null,
            "AboveGradeFinishedAreaSource": null,
            "AboveGradeFinishedAreaUnits": null,
            "...": "..."
         }]
}

Code Example

https://api-prod.corelogic.com/trestle/RETS/Search?SearchType=Property&Class=Property&Query=(ListPrice = 1000000)&QueryType=DMQL2&Format=COMPACT-DECODED&Limit=1000&StandardNames=0&Select=ListingKey&Orderby=ListingKey&Count=1&Replication=true

Result

<RETS ReplyCode="0" ReplyText="Success.">
    <COUNT Records="668" />
    <DELIMITER value="09" />
    <NEXTLINK>https://api-prod.corelogic.com/trestle/RETS/Replication?id=123-a119257f-5dc0-4a47-bbb3-e526efd22776</NEXTLINK>
    <COLUMNS>	ListingKey	</COLUMNS>
    <DATA>	999679787	</DATA>
    <DATA>	999668656	</DATA>
    <DATA>	999662434	</DATA>
    <DATA>	362654651	</DATA>
    <MAXROWS />
</RETS>