Repository navigation
FreeBSD UFS2 file per directory limit #13091
Description
Steps to reproduce
- Install FreeBSD with default partition type (UFS)
- Install nextcloud
- Upload your files
- Generate previews with previewgenerator app
Expected behaviour
All previews should be generated
Actual behaviour
Only first 32765 previews are generated
Server configuration
Operating system: FreeBSD 12
Web server: nginx
Database: MariaDB 10.3
PHP version: 7.1
Nextcloud version: 15.0.0
Updated from an older Nextcloud/ownCloud or fresh install: Updated
Where did you install Nextcloud from: Official repository
Signing status:
Signing status
No errors have been found.
List of activated apps:
App list
Enabled: - accessibility: 1.1.0 - activity: 2.8.2 - bruteforcesettings: 1.2.0 - calendar: 1.6.4 - cloud_federation_api: 0.1.0 - comments: 1.5.0 - contacts: 3.0.0 - dav: 1.8.0 - federatedfilesharing: 1.5.0 - federation: 1.5.0 - files: 1.10.0 - files_pdfviewer: 1.4.0 - files_sharing: 1.7.0 - files_texteditor: 2.7.0 - files_trashbin: 1.5.0 - files_versions: 1.8.0 - files_videoplayer: 1.4.0 - firstrunwizard: 2.4.0 - gallery: 18.2.0 - logreader: 2.0.0 - lookup_server_connector: 1.3.0 - nextcloud_announcements: 1.4.0 - notifications: 2.3.0 - oauth2: 1.3.0 - password_policy: 1.5.0 - provisioning_api: 1.5.0 - serverinfo: 1.5.0 - sharebymail: 1.5.0 - support: 1.0.0 - survey_client: 1.3.0 - systemtags: 1.5.0 - theming: 1.6.0 - twofactor_backupcodes: 1.4.1 - updatenotification: 1.5.0 - workflowengine: 1.5.0 Disabled: - admin_audit - encryption - files_accesscontrol - files_antivirus - files_external - user_external - user_ldapNextcloud configuration:
Config report
"system": { "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "localhost", ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "overwrite.cli.url": "http:\/\/localhost", "dbtype": "mysql", "version": "15.0.0.10", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "mysql.utf8mb4": true, "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "filelocking.enabled": false, "instanceid": "***REMOVED SENSITIVE VALUE***", "auth.bruteforce.protection.enabled": false, "maintenance": false, "theme": "", "loglevel": 0, "updater.release.channel": "stable" }Are you using external storage, if yes which one: no
Are you using encryption: no
Are you using an external user-backend, if yes which one: no
Client configuration
Browser: Safari
Operating system: MacOS
Logs
Nextcloud log (data/nextcloud.log)
Nextcloud log
{"reqId":"FVn52UwMTvg8lwzAErGz","level":0,"time":"2018-12-15T19:02:31+00:00","remoteAddr":"X.X.X.X","user":"XXX","app":"PHP","method":"GET","url":"/nextcloud/index.php/core/preview.png?file=text.txt&x=281&y=500&a=1&mode=cover","message":"mkdir(): Too many links at /usr/local/www/nextcloud/lib/private/Files/Storage/Local.php#85","userAgent":"Mozilla/5.0 (iOS) Nextcloud-iOS/2.22.7","version":"15.0.0.10"}
{"reqId":"4dvrvLb2Ei1q7J5NUf9H","level":0,"time":"2018-12-15T19:04:35+00:00","remoteAddr":"X.X.X.X","user":"XXX","app":"PHP","method":"GET","url":"/nextcloud/index.php/core/preview.png?file=text.txt&x=281&y=500&a=1&mode=cover","message":"mkdir(): Too many links at /usr/local/www/nextcloud/lib/private/Files/Storage/Local.php#85","userAgent":"Mozilla/5.0 (iOS) Nextcloud-iOS/2.22.7","version":"15.0.0.10"}
Activity
giraudeo commentedon Dec 16, 2018
Hello,
First of all, just a big thanks to all the Nextcloud team for the hard work you are providing to the open-source community. I am using Nextcloud for 1 year and half for personal usage, and I am very happy with it up to now ;)
So, to summarise my bug, I spent a good part of yesterday to see why and where is coming this UFS2 limitation.
Best summary would be this thread:
https://lists.freebsd.org/pipermail/freebsd-fs/2005-March/001001.html
I am not a FS expert, but for what I understand, the default inode table on UFS2 is made to accept only 32767 references.
Different workarounds would be to modify this inode limit, or to move to a different FS (or OS).
Right now, I do not have really this possibility, and was wondering if something could be done on the application level?
For my defence, I found other discussions where it is not recommended to go with large amount of files inside a directory.
like here:
https://forums.freebsd.org/threads/how-many-files-could-be-stored-in-a-directory.48787/
Thx again!
Regards,
Olivier
tflidd commentedon Dec 17, 2018
Perhaps you could program an app that distributes the files over several folders but shows it as a single folder in the interface. Many discourage to use too many files in a single folder: https://stackoverflow.com/questions/466521/how-many-files-can-i-put-in-a-directory
Is there still a specific bug in Nextcloud you are referring to? If it's more about how to realize your setup, you might be better off in the forum help.nextcloud.com
giraudeo commentedon Dec 17, 2018
Hey tflidd,
Thanks for your reply.
The problem is coming from the flat structure of the Nextcloud preview generation itself.
All previews are stored in next cloud/data/appdata_XXXXXX/preview, and it reaches this 32K files limit:
root@localhost:/usr/local/www/nextcloud/data/appdata_XXXXXX/preview # mkdir mytest
mkdir: mytest: Too many links
root@localhost:/usr/local/www/nextcloud/data/appdata_XXXXXX/preview # ls -a | wc -l
32767
Let me know if you need more details.
Regards,
Olivier
tflidd commentedon Dec 17, 2018
Thanks for clarification. Could perhaps increase the performance on certain systems.
@rullzer
gbmaster commentedon Aug 14, 2019
The bug is also present in OpenBSD using FFS.
Running
ls -a | wc -l
returns 32767
skjnldsv commentedon Sep 9, 2020
Closing as wontfix, please increase the number of available inodes. For me this is an issue outside of NC