How scalable is this bookmarks app? Can it handle 1 million+ links?

[flamingjupiter]

Title is self-explanatory. I'm searching for a solution that's scalable. I know performance will depend on the host machine specs, so imagine a high end, last gen domestic rig running Nextcloud and bookmarks app. Would it be able to handle that many links and, say, 10k tags?

[marcelklehr]

Hey!

I haven't done intensive testing on scalabiltiy so far. I know @muppeth were running a big instance in the past. Not sure if you're still running that. Maybe you can shed some light on this :)

[flamingjupiter]

Heh, just reread my comment and realized I didn't even say hi. Sorry for my rudeness, I typed it in a rush

[marcelklehr]

Did you give it whirl to test this? I think an important aspect for this is how many links there are per user.

[marcelklehr]

@flamingjupiter If you have tried this in a scenario with lots of links, I would be interested in your findings. If you have trouble making the app scale, we could look into it together

[marcelklehr]

@muppeth Are you still admin of a big instance of nextcloud with the bookmarks installed? Would be interested in your experiences :)

[marcelklehr]

So, I have data from an instance that has ~1500 users and ~500000 links which works smoothly.

[flamingjupiter]

Hey man, sorry for my absence. I still couldn't test it with a big instance, I'm planning to do it in the next few months. I'm thrilled by your findings. I'd be the only user of my big instance, didn't know the amount of users would be relevant for performance. As soon as I test it I will let you know, thank you very much for the attention.

Should I check your reply as "mark as answer"?

[sproggit]

I'm sorry this is only half an answer... I've just checked my browser, which is syncing perfectly, and my bookmark HTML export runs to about 7Mb on disc. I appreciate how imprecise that is, given the variable nature of the file. If someone can suggest a link-specific XML tag, I could load my bookmarks file in to e.g. a Word Processor and "Find" the tag - it should be easy to get a count that way.

I would just also like to add that if you really need a million bookmarks, then it may well be that your bottleneck could become your browser rather than this solution. I know that Firefox uses SQLite for instance, which in theory can easily handle that many bookmarks... although performance guidance for large tables with that product carry some interesting warnings about performance drop-off if you have inefficient tables, or if you get your indexing wrong, or if you don't keep your table clean [as in, the need to avoid disjointed index splits].

Of course, this gives the question of how to practically test.

It must be possible to synthetically create a URL with a general form like this:-

      https://www.yourtestserver.com/index.php?{an incrementing 8-digit number}

You can create a textfile with that pattern and 10 million rows with a couple of lines of python... The next part of the problem would be to get this in to a web browser's bookmarks repository. If there is no easy way to do so by simply directly writing to the browser's SQLite instance, maybe you could set up Apache JMeter to automate it for you? [Sorry, not sufficiently familiar with JMeter myself...]

But once you've got the URL's loaded in to a browser, proving/disproving the scalability should be pretty straightforward.

[marcelklehr]

Thanks for weighing in, the thing is, Nextcloud allows multiple users, so even if all users have only 1k bookmarks, the overall amount might go into the millions. That's what this discussion is about, I believe.

[sproggit]

Fair - and thanks for responding. But I still think it should be possible to test this experimentally and with relatively little effort...

[marcelklehr]

Indeed, it should be possible to test this. If youre adventurous you could even be the one who does it :)