Technical Explanation

In the past we've tried this by rolling our own mongodb system (you have to have multiple systems because mongodb has a failsafe) which is expensive, and I'm honestly not a system administrator well versed in running massive database systems. So in those cases, it didn't really work.

Now, we're using a database as a service (compose.io) to host this information. A service that is far better at running mongodb (its their job!) and can help support us in the case that we have a problem with our data.

We're looking at doing this for some of other "big data" type information which includes user logs. We're watching how this goes with the price information, which is a lot of information. When I flipped the switch this morning almost 8,000 price points were added immediately. We're up to 18,000 now about an hour later.

If it goes well, we'll extend it to logs which will make UAs helping a lot easier for most users.

Yup! Account worth (and sorting by price, etc) is all tied to the same system, so after a few days it should be a lot more accurate.

It should! I haven't pushed these changes to that page yet (I want to give it a few days to collect information) but it should help that page a lot once I do.