Rob Thorpe (rthorpe@realworldtech.com) on 2/2/09 wrote:
---------------------------
>Dean Kent (dkent@realworldtech.com) on 1/31/09 wrote:
>---------------------------
>>rwessel (robertwessel@yahoo.com) on 1/31/09 wrote:
>>---------------------------
>>
>>>
>>>But your approach will certainly have scaling issues.
>>
>>As a writer of commercial software, this is one of the most common problems I see
>>in any long-lived product. It seems that the average person (meaning, average developer/programmer)
>>has trouble keeping in mind all of the various pieces of a design that will need
>>to scale as the number of resources used or tracked increases.
>>
>>And, it is one of the most difficult to get people to see without concrete examples.
>>There have been many an argument between developers that only gets settled after
>>many years of a product in the field. And all too often those who should be seeing
>>the results to learn from either no longer remember (or care), or are no longer around to do so.
>>
>>Just my observation.
>
>Yes.
>
>I've made the same mistake. I used a text file for storing something that eventually
>turned out to require many MB of data. It made the program slow to a crawl at times.
>Naturally it made me a bit unpopular.


Or you can have a customer make that mistake, and then it becomes your problem. One of our products can use a database of information to manage requests from clients. There is support for using a real database, a user written plug-in, a user written application on a different machine, and a simple text file.

The text file was intended to support testing and simple installations with modest and static requirements, and while the text file database was loaded into memory, access was single threaded, and the code was simple (IOW, sequential searches, etc.). All perfectly reasonable for a few hundred entries, much less the "practical" limit of a few dozen entries we ever expected to see (and the likely limit of perhaps a couple dozen).

Needless to say, a customer decided that they wanted to use their existing database, but not write a plug-in to access it, so they wrote a little export procedure to create a text file. 50K+ entries later, load time (and reload time - every change required them to copy the recreated text file to the right place and ask for a reload) was excessive, and since it was single threaded, prevented sessions from starting for the duration. Lookups (sequential searches through the 50K entries) were taking a significant amount of the available CPU time. The storage format in memory was also not very compact (for a few hundred entries, who cares, right?), so the loaded database was taking on the order of .75GB of memory.

Of course, this was now our problem. After discussions with the customer, they really wanted (or at least that was the only politically acceptable solution) to keep their text file, so we improved that, loading the new database file while keeping the old database active to serve requests, faster searches, more compact storage...

Scaling problem often pop up in unexpected places, even in places where you "know" it won’t be an issue.