PHP mysql connections

  • SpooF
  • ٩๏̯͡๏۶
  • Bronze Member
  • User avatar
  • Posts: 3423
  • Loc: Richland, WA

Post 3+ Months Ago

One of the projects I'm currently working on requires a massive mount of data collection. Our goal is to reach 20,000 requests an hour, so around 330 requests per minute (5.4 per second). On each request we are downloading ~37 kb of data (xml), each request has 20 independent blocks, and we are extracting the parts we want from each block. Each block is a dataset so we have 20 inserts to the database per request. Thats 108 inserts to the database every second. At a rate like this would it be faster to combine all of those 20 calls into a single query? I'm not sure if that will really make it faster or not. I'm pretty sure the most of the problems people run into are making connections to a database. Which is my next question.

To get to the 20,000 requests an hour I'm thinking of making a long term process to do all this work. That way as soon as one request is completed the next one is started and there is no downtime while we wait for the cron to come along and start the php script going again. This would also stop us from making a new connection to the database every time we make a request. Instead there would be a single connection.

I'm thinking of writing the long term processes in python. Do you think it would be a good choice or should I do it in c/c++?

Also this is just our data collection this doesn't not include post processing or client requests to the database.

I'm not even sure if this is possible at the moment to do from a single server. I might have to setup a few to split the work between. I've been running a few tests using php, and to just download the content and phrase the xml into an object is taking quite a bit of time.

Post Information

  • Total Posts in this topic: 1 post
  • Users browsing this forum: No registered users and 27 guests
  • You cannot post new topics in this forum
  • You cannot reply to topics in this forum
  • You cannot edit your posts in this forum
  • You cannot delete your posts in this forum
  • You cannot post attachments in this forum

© 1998-2017. Ozzu® is a registered trademark of Unmelted, LLC.