Post Reply 
 
Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Very Large Collection
06-05-2018, 07:56 PM
Post: #1
Very Large Collection
Hi,

I just ran across Recollector the other day, have downloaded the demo and have been playing around with it. The program looks to be what I'm looking for, but have a couple of questions before I purchase.

My Dad has a matchbook cover collection that he started when he was a child back in the late 1930's. He's amassed nearly 1 million different covers. Up until this point, he's got no database of his collection. He's got them housed in over 700 albums (3 ring binders).

He's a general collector, which means he collects matchbook covers from any category. I'd guess he's got them broken down into 60 or so main categories. Some categories may have 3 or 4 thousand covers in them, others may have only a few hundred. BUT, many could be categorized in MULTIPLE categories.

I'd like to be able to have his entire collection set up in a single database so that we can run multiple queries (subsets?) at a time. Say for example, if we want to see a list of Diamond Quality matchcovers that feature truck lines from Iowa that are not struck. Or girlie covers from Massachusetts that are from 1940 that advertise an auto dealer. Or all the matchbook covers in his collection from Idaho. Am I correct in my thinking that in order to do this, we'd have to have a single database? Or can multiple databases (collections) be queried at the same time, generating a single list? Or would I need to query each database and then have to combine multiple lists?

I read where a database can be as large as you want it to be, but that the larger it is, the longer it's going to take to access when first starting the program. Understood. I would also figure that I'd be opening us up for a far greater chance of running into a corrupt file along the way if we've got several hundred thousand entries in it.

Thanks for any thoughts on how to set this up.

Warren
Find all posts by this user
Quote this message in a reply
06-06-2018, 12:16 AM
Post: #2
RE: Very Large Collection
Thanks for the inquiry.

You definitely would be pushing into unexplored territory if you create a collection with hundreds of thousands of records. It is true that there is no formal limit to the number of records allowed. (Compare, for example, Excel, which - in earlier versions - had a specific hard limit of 65,536 rows, and - in the later versions - has a hard limit of 1,048,576 rows.)

Recollector does read the entire collection into memory, so if you have a HUGE collection, then you may suffer the consequences of the program using large amounts of operating system memory. Those consequences are usually just a matter of slow-downs in processing, as the virtual memory system swaps pages of memory to and from disk. But you could also eventually bump up against some operating-system limit of the size of virtual memory allowed for any one running program.

We have run tests on collections with tens of thousands of records, and these don't show much degradation in performance on machines with adequate memory and processing power. But collections with hundreds of thousands of records haven't been tested.

There is not currently any facility for searching across multiple collections. This is a request which has been made by other users, and I will be looking into whether this makes sense as a future enhancement.

The elephant in the closet, though, is getting the data entered in the first place. You say that you currently have no computerized database; just a set of 700 or so three-ring binders. Doing data-entry on a collection of hundreds of thousands of items is an imposing challenge! If you estimate that it would take 1 minute to enter the data for each item, that's 1,666 hours of data entry (or over 200 8-hour non-stop days) for the first 100,000 items! And will you be photographing each cover and attaching the image to the record? If so, that's more time and work.

I don't mean to discourage your moving forward on this project, but I think you have some issues of scale that need to be reckoned with, and better to think them through beforehand than spend a lot of time getting part of the way down the road and then run out of steam.
Find all posts by this user
Quote this message in a reply
06-06-2018, 02:09 AM
Post: #3
RE: Very Large Collection
Thanks for your reply and thoughts on this project. Indeed, plenty to think of here.
Find all posts by this user
Quote this message in a reply
06-06-2018, 11:46 PM (This post was last modified: 06-06-2018 11:48 PM by warbar.)
Post: #4
RE: Very Large Collection
This is a project I'm going to proceed with and do understand it's likely to take close to a decade to complete. I've scanned about 40% of the matchcovers so far - and it's taken close to 5 years to do so. Not a daily task, of course, but here and there. The kicker with this is they are all stored in plastic pages - 16 to a page - 8 back to back. Problem is the pockets are too large and the covers are at all different angles. Autocropping / straightening scripts don't work because the matchcovers in the front and the back don't always line up with each other, so the script doesn't know which one to straighten / crop. So, all the cropping / straightening has to be done by hand.

I've looked at a number of programs to get the collection into a database this task and keep coming back to this one. No way I'm going to do this as a single database. Way to large and way too much of a potential to end up with a corrupted file down the line. The only hang up right now is the inability to query between databases. But if you think this is functionality that you would be willing to add at some point, then having 60+ databases wouldn't be an issue for me.

Thanks again.

Warren
Find all posts by this user
Quote this message in a reply
06-07-2018, 08:08 AM
Post: #5
RE: Very Large Collection
You are really undertaking an impressive task!

I do think it would be a safe enough bet to use Recollector to get started, and perhaps all the way through. Perhaps a first goal would be to build the first (of your estimated 60+) collections, just to see how much work is involved in doing this, and to convince yourself that Recollector gives you access to the information in a way that is good for you.

One thing that should give you some comfort is the fact that the data that you put into Recollector can be easily exported, at some point in the future, if you come to realize that you have found a better solution for your mammoth collection. Exporting to Excel or to comma-separated value (CSV) files is a built-in feature of the product, so whatever data-entry you do in Recollector, you should feel confident that you won't have to do that data-entry again, even if you eventually decide to migrate to a different solution.

And I will be exploring the cross-collection searching capability that we have been discussing.
Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump:


User(s) browsing this thread: 1 Guest(s)