10,000 Ancestors Limit

Craig Andrew Miles PRO

3/12/2023 at 3:27 AM

There appears to be a limit of 10,000 to the list of ancestors returned using https://www.geni.com/list?group=ancestors

Is there any way around this limit?

I tried applying a filter of Gender=Male seeking to extend the reach, but unfortunately the filter is applied after the 10,000 ancestor limit is enforced and I only get around 5,000 of the Male ancestors from the full list of 10,000.

This makes seeing beyond the 10,000 limit, in my case 31 generations, impossible in Geni.

I understand the rationale behind the limitation in terms of computational power to evaluate an ever increasing ancestry depth, and to be honest I am impressed that Geni does so well at this, particularly when compared to its sister site MyHeritage that seems to really struggle with a depth of generations, but if you have a deep tree the 10,000 limit is a very hard boundary to come up against.

I could do something programmatically via the API I guess, different to what I have already done, but it would be nice if there was some kind of workaround to get deeper than the 10,000 limit using the Geni UI.

Any thoughts, anyone?

Private User

3/12/2023 at 9:01 AM

Well, since you are posting in the The Geni API project the answer should be obvious: Use the API to traverse the ancestor generations yourself instead of using the web-interface as you do. I have done it myself.

Just remember to keep a track of which profiles you already have processed to avoid re-entering the parents of a profile you already have processed in another line - the further back in generations you get, the more ancestors they have in common.

Craig Andrew Miles PRO

3/12/2023 at 3:49 PM

Well of course you are correct Private User

But this approach requires a starting point of a given generation in order to evaluate the next generation. This is fine, but you can never be sure that something hasnt shifted since the previous generation was determined, so the only 100% reliable approach is to start at generation 1 each time.

With the API rate limit in place this requires a minimum of 10,000 seconds just to get to the starting point of 10,000 ancestors each time!

Unless you know of a way of deriving a given generation of ancestors (number of steps) from a given starting point using the API? But I cant see anything in the documentation that looks like this?

Private User

3/14/2023 at 4:13 AM

I did one generation at the time, and stored processed profiles in a Dictionary with Profile-Id as key to store and check if a profile already is entered. For each generation I stacked up immediate-family request ID's which I then bulk-loaded 50 of at the time using the ids=xx,yy,zz parameter, and using the fields parameter to reduce the size of the request to only get the fields I needed for further processing, for example: fields=id,name,gender&only_ids=true..
If you are only traversing the tree and are only interested in the name and connection you does not have to make a separate request for the profiles, -you can request the profile details when needed (and of course use bulk-loading of them as well).

Hitting the API rate causes no problem because I made a detection for that, making a re-request after suspending the thread a bit. You can also slow down requests a bit when getting close to the rate-limit since you can monitor rate limit since rate limit values is returned in the request headers.

This is how I show Api Rate in a desktop app displaying a bar with (X-API-Rate-Limit - X-API-Rate-Remaining), - i.e the more red the less requests you have left in the X-API-Rate-Window:
www.geni.com/media/proxy?media_id=6000000192743245821&size=large

Private User

3/14/2023 at 4:19 AM

Unless you missed the hint: Api-Rate is number of requests within a time-frame, independent of how much you ask for in one request, - in the above example I got 50 records in one request, one page at the time if you requested more than 50.

Private User

3/14/2023 at 4:20 AM

But, - Be careful: If you choke the Geni servers the Api limitations might change.

Craig Andrew Miles PRO

3/14/2023 at 10:52 PM

Yes, thanks for all of the good advice Private User.

I have had to deal with the API Rate Limit already and have to throttle back execution with a sleep.

I was hoping though that Mike Stangel might have some workaround to wrangle the UI to being more accommodating to more than 10,000 ancestors seeing as how my home page seems to have no problem evaluating the amount I actually have (17K+).

Private User

3/15/2023 at 9:39 AM

Well, again you ask about something that has nothing to do with the API because there is no 10,000 limit in the API, only in the web interface.
Earlier, before they got better servers the limit in the Web interface was much lower.

Craig Andrew Miles PRO

3/15/2023 at 3:19 PM

Private User

Ha! Not strictly to do with the API, no, if you want to be pedantic, but on the gneral topic of the discussion.

Since you ask, this is the only forum on which I seem to get any kind of response - my 10,000 ancestors feature request has hone unanswered since January 2021

https://help.geni.com/hc/en-us/community/posts/360051663254-Ancesto...

I havent figured out what the official engagement policy is by Geni to feature and bug requests - unofficially they seem to pretty much ignore everything and hope that enthusiasts such as yourself jump in.

Private User

3/16/2023 at 4:26 AM

Well, if you look at that URL you see the part community/posts, which means that it is a group where the community, not Geni staff tries to help each other, but most prefer to just use the public discussions, especially now when you can include pictures as examples, - as I did above.

Only one have voted up that request, and I guess that it is yourself, but it gives you an indication of the activity level.

Even worse in the Geni Wiki pages where there has been almost no activity the last years.

Log in using Email

10,000 Ancestors Limit

Participants:

Related Projects:

Create a free account or login to participate in this discussion