Optimizing PT performance for BIG databases...remote database?

[Entity]

Bit of a long story, I'm going to try to make it short:

I'm tired of PT taking what feels like hours to import a few thousand hands, when I'm datamining, on a top-notch rig (AMD X2 Dual 4400+, 2gb RAM, SATA 7200RPM drive, blah blah blah). Generally I'm mining a fairly large number of tables distributed across a few PCs with VMWare running, but sometimes PT just seems to take a ridiculous amount of time to import the hands. When it's doing this, the PC is basically usable for web browsing, AIM, but that's about it; I can't do much in the way of poker. Ideally I'd like to try to get my home computer setup so that I can continue to mine on some tables while playing on another -- so I've decided to move the DB off to one of my other computers. Anyone currently doing this? I'd like to streamline it as much as possible.

Here are the setups that I have:

For playing:

XP 4400+, Dual Core, 2gb RAM (DDR400/PC3200, soon to be 4gb), 2x 7200RPM SATA HDS (250gb and 320gb). DFI Lanparty UT-SLI NForce4 board. Connected to LAN via a gigabit card and Netgear GS105 gigabet switch. System seems to scream for everything except PT. Ideally I will be running PAHud and Party on this machine. I also frequently run multiple instances of VMWare either to datamine or to work on other projects where I need multiple clients of XP running.

Setup #2:

Older Northwood core P4 3.06ghz. 2gb RAM (PC2100). MSI 845PEMax2 motherboard, Adaptec 29160N U160 SCSI controller + Fujitsu Ultra320 15kRPM HD (36.7gb). Also has an older 60 or 80GB Deskstar HD (7200RPM) that I wouldn't mind using as a backup drive for the DBs / installation drive.

Since I'm effectively trying to start from scratch here, I'd appreciate opinions on the best way to set everything up, given that I intend on having multiple million+hand DBs being stored on the computer. I'm considering running PT on both computers, but only running the observed DB on one of them; any sense to doing this? One of the reasons I'm doing this now is that I plan on getting a lot of data stored, and am hoping to do at least some analysis on it later on (mostly blind defense and things like that).

Anyway. Long story, tried to make it short, mostly failed. Would appreciate any comments before I plunge headfirst into this.

Rob

[Percula]

I guess the big question is rather you need to run PT on the machine you are playing on or not. My guess it that it is not required.

If you can, then I would setup something like... FC5 on the older PC with PSQL, Samba and VMWare server. Setup a VM with XP for PT. Setup your poker apps to save HH to a Samba share, point PT from the other VM to that same share and to use PSQL on the FC5 box.

I am just not sure about PAHUD and what exactly you would need to do there. If you can just install the PSQL ODBC and ODBC DSN you should be good. Maybe just install PT on your poker manchine and get it setup to use the PSQL DB's and that should be enough for PAHUD and just never use the PT install on the poker machine. You would have to play with it, at PAHUD folks etc it will work without PT installed locally or not.

At least that way, you wouldn't be affected by the PT lag of importing hands.

[bhudson]

[ QUOTE ]
When it's doing this, the PC is basically usable for web browsing, AIM, but that's about it; I can't do much in the way of poker.

[/ QUOTE ]

I have a very similar configuration to yours and I don't see this. CPU usage tops out at 40% (mostly postgreSQL). Something wrong here.

Remote DB should be fine for analysis.

[Entity]

[ QUOTE ]
[ QUOTE ]
When it's doing this, the PC is basically usable for web browsing, AIM, but that's about it; I can't do much in the way of poker.

[/ QUOTE ]

I have a very similar configuration to yours and I don't see this. CPU usage tops out at 40% (mostly postgreSQL). Something wrong here.

Remote DB should be fine for analysis.

[/ QUOTE ]

How many hands are you importing when this is the case? It's not that way for nonobserved hands, but when I'm mining, it's usually pulling about ~6k hands per hour.

[piki]

Try reducing Poker Tracker's priority to below normal (ctrl-alt-delete, processes, right click ptrack2.exe, set priority).

Btw, I have database on remote machine, but it doesn't offer any speedups during import. I use monthly databases (each about 1mio mined hands) for observed hands to keep the imports reasonably fast. To avoid constantly rebuilding PAHud's cache, I've created monthly databases in advance (April-December) and included them all.

-pix

[bhudson]

Hard to say. I'd estimate around 2K/hr, could be more. I mean, it's totally a non-issue for me, I wouldn't even know it's there.

Since you have everything spread over multiple PCs, it could be a network problem especially if your CPU usage is not spiking under task manager. I use NetBEUI, not TCP/IP.

[APerfect10]

With your proposed setup I would do the following: (I've been thinking about doing this for awhile now)

1. Move your Postgres databases to your server.
2. Increase Postgres' memory usages to about 75% of your server's memory.
2. Install PT on both server & main computer
3. Use PT on server for import; PT on main computer only for viewing stats

This would alleviate PT's and Postgres' CPU load on your main computer. On my machine, I'm looking at ~50-60% CPU usage at times.

You may notice a slight speed-up of imports with this situation. Probably nothing very notable if at all. The key IMHO is the reduced CPU load on your main computer...

[Fishy McDonk]

To resolve this:

[ QUOTE ]
When it's doing this, the PC is basically usable for web browsing, AIM, but that's about it; I can't do much in the way of poker.

[/ QUOTE ]

do this:

[ QUOTE ]
Try reducing Poker Tracker's priority to below normal (ctrl-alt-delete, processes, right click ptrack2.exe, set priority).

[/ QUOTE ]

[jukofyork]

I think the main problem with PT is that their is some fundamental problem with it's design (either in it's schema or in the use of PowerBuilder code for it's parsers).

I agree that when PT was first designed then the sheer amount of data generated by datamining was never foreseen, but there seems to be very little work aimed at improving this aspect over the last 2 years.

The PAHUD and SS meta databases seem to be able to get round the slowness of retrieval, but by far the most frustrating part is the time it takes to import the hands (plus the strange exponential performance decay).

I have several million hand histories now collected and cannot face importing them into PT because of the stupid amount of time it would take to do this. The last time I tried, I borrowed a cutting edge machine (with raid drives and extreme amounts of memory) and it took something like 3 days to import 700k hands into a PGSQL DB. Then when I moved it back to my machine it was totally unusable and I had to give in (pre meta databases).

I then moved to using PokerManager. Since using PM I quite happily imported and used a 3 million hand DB without any of the problems I had with PT (no exponential import times, no machine lockups for 40 mins, no unusable DBs, overall 3-4x faster imports, smaller DB size, etc).

A) If more people were to look into using PM then I'm sure it would in turn become more popular and Ben (the SS author) would be more willing to work on it and update/improve it. Ultimately we would all benefit from this, yet everybody seem to be happy to keep using PT with all of the flaws.

B) Alternatively (if it is within copyright law), I am sure somebody could use the PT schema (if it is not the schema itself which is flawed...) and write much faster parsers for getting the data into the DB in the first place.

C) If outside of copyright law or the schema itself is fatally flawed, then perhaps somebody could sit down and completely redesign a "database" or schema. I'm not even convinced that using commercially available DBs is necessary and just creates unnecessary bloat.

Poker Tracker may have cornered the market when it comes to personal hand history analysis, but I feel there is a HUGE market (of frustrated PT users) just waiting for a decent solution to observed hand history import and retrieval.

Juk [img]/images/graemlins/smile.gif[/img]

[Entity]

[ QUOTE ]
I think the main problem with PT is that their is some fundamental problem with it's design (either in it's schema or in the use of PowerBuilder code for it's parsers).

I agree that when PT was first designed then the sheer amount of data generated by datamining was never foreseen, but there seems to be very little work aimed at improving this aspect over the last 2 years.

The PAHUD and SS meta databases seem to be able to get round the slowness of retrieval, but by far the most frustrating part is the time it takes to import the hands (plus the strange exponential performance decay).

I have several million hand histories now collected and cannot face importing them into PT because of the stupid amount of time it would take to do this. The last time I tried, I borrowed a cutting edge machine (with raid drives and extreme amounts of memory) and it took something like 3 days to import 700k hands into a PGSQL DB. Then when I moved it back to my machine it was totally unusable and I had to give in (pre meta databases).

I then moved to using PokerManager. Since using PM I quite happily imported and used a 3 million hand DB without any of the problems I had with PT (no exponential import times, no machine lockups for 40 mins, no unusable DBs, overall 3-4x faster imports, smaller DB size, etc).

A) If more people were to look into using PM then I'm sure it would in turn become more popular and Ben (the SS author) would be more willing to work on it and update/improve it. Ultimately we would all benefit from this, yet everybody seem to be happy to keep using PT with all of the flaws.

B) Alternatively (if it is within copyright law), I am sure somebody could use the PT schema (if it is not the schema itself which is flawed...) and write much faster parsers for getting the data into the DB in the first place.

C) If outside of copyright law or the schema itself is fatally flawed, then perhaps somebody could sit down and completely redesign a "database" or schema. I'm not even convinced that using commercially available DBs is necessary and just creates unnecessary bloat.

Poker Tracker may have cornered the market when it comes to personal hand history analysis, but I feel there is a HUGE market (of frustrated PT users) just waiting for a decent solution to observed hand history import and retrieval.

Juk [img]/images/graemlins/smile.gif[/img]

[/ QUOTE ]

PM looks good; I'll keep it in mind for some use in some of the analysis I hope to do on the larger databases. Do you know if there are any plans for any HUDs in the future to use/support it?

Rob