Optimizing PT performance for BIG databases...remote database?

[jukofyork]

[ QUOTE ]
[ QUOTE ]
I think the main problem with PT is that their is some fundamental problem with it's design (either in it's schema or in the use of PowerBuilder code for it's parsers).

I agree that when PT was first designed then the sheer amount of data generated by datamining was never foreseen, but there seems to be very little work aimed at improving this aspect over the last 2 years.

The PAHUD and SS meta databases seem to be able to get round the slowness of retrieval, but by far the most frustrating part is the time it takes to import the hands (plus the strange exponential performance decay).

I have several million hand histories now collected and cannot face importing them into PT because of the stupid amount of time it would take to do this. The last time I tried, I borrowed a cutting edge machine (with raid drives and extreme amounts of memory) and it took something like 3 days to import 700k hands into a PGSQL DB. Then when I moved it back to my machine it was totally unusable and I had to give in (pre meta databases).

I then moved to using PokerManager. Since using PM I quite happily imported and used a 3 million hand DB without any of the problems I had with PT (no exponential import times, no machine lockups for 40 mins, no unusable DBs, overall 3-4x faster imports, smaller DB size, etc).

A) If more people were to look into using PM then I'm sure it would in turn become more popular and Ben (the SS author) would be more willing to work on it and update/improve it. Ultimately we would all benefit from this, yet everybody seem to be happy to keep using PT with all of the flaws.

B) Alternatively (if it is within copyright law), I am sure somebody could use the PT schema (if it is not the schema itself which is flawed...) and write much faster parsers for getting the data into the DB in the first place.

C) If outside of copyright law or the schema itself is fatally flawed, then perhaps somebody could sit down and completely redesign a "database" or schema. I'm not even convinced that using commercially available DBs is necessary and just creates unnecessary bloat.

Poker Tracker may have cornered the market when it comes to personal hand history analysis, but I feel there is a HUGE market (of frustrated PT users) just waiting for a decent solution to observed hand history import and retrieval.

Juk [img]/images/graemlins/smile.gif[/img]

[/ QUOTE ]

PM looks good; I'll keep it in mind for some use in some of the analysis I hope to do on the larger databases. Do you know if there are any plans for any HUDs in the future to use/support it?

[/ QUOTE ]
I think it comes down to the fact that Ben spent ages getting it to the stage where it is now, but there was just an overall lack of interest in it (I agree it needs polishing and can be improved on, but on the other hand I can understand that if their is no real PM user-base then their is no incentive to do this...).

I looked into it and did manage to figure out how to query the FireBird DB from C/C++ and it looks like it wouldn't actually be that hard to get GT+ to use the PM DB rather than the PT DB (I have some code snippets that will let me send queries to the DB and Ben published a list of helpful queries in the PM forum to help).

I think it's a chicken and egg problem. Everything supports PT because it got in their first, so they are tied to using PT. Nobody wants to give up their tools which rely on PT, but I'm pretty some nearly everybody would appreciate being able to handle much more data without all the problems associated with PT.

Juk [img]/images/graemlins/smile.gif[/img]