Plans and ideas for wxSQLite3 version 5.x

[utelle]

Work on the next major version 5.x of wxSQLite3 will start shortly. There are already a few plans (which will definitely be fulfilled) and ideas (which may or may not be fulfilled). However, I would like to get feedback from the wxSQLite3 user base, before making final decisions.

Please feel free to comment on the following list and/or complement it with additional ideas.

1. Plans

   • Add C++ iterator support for handling result sets
   • Drop support for dynamically loading the SQLite library
   • Drop support for wxWidgets 2.x

2. Ideas

   • Introduce namespace, i.e. wxSQLite3, instead of class name prefix (for example, class wxSQLite3Database would become wxSQLite3::Database)
   • Derive exceptions from std::exception (or std::runtime_error)
   • Make use of C++11 features where appropriate
   • Make SQLite Encryption Extension available as a separate GitHub project

Implementing ideas would break source code compatibility with previous versions in most, if not all cases. Therefore the question is whether keeping up a 4.x compatibility branch would be desirable.

Feedback is welcome.

[utelle]

Almost 4 weeks have passed since I opened this issue, but obviously no one seems to care about incompatible changes in version 5.x. So ... let's wait and see what's going to happen in version 5.x.

[Willena]

In my case, I'm more interested by the fact that the SQLite Encryption Extension will now become a separate repository because I'm only using the SEE to build a Java wrapper.

[utelle]

I'm working on the separation of the encryption extension. Probably a first release will be done within September, but there is not yet a fixed date.

[sndnvaps]

i have separation the encryption sqlite https://github.com/sndnvaps/SQLite3-Encryption

[utelle]

@sndnvaps: Thanks for mentioning this.

However, you forked your repository from rindeal/SQLite3-Encryption, which was not updated since 2016 and is now archived. Yes, I see that you spent some time to improve your fork, and of course I will take a closer look.

Nevertheless, I intend to set up my own repository, because to me this seems to be the more natural approach. In fact, it's already there, but currently still empty. This will change within this month.

[slodki]

@utelle Will the SQLite Encryption Extension separation allow to build dynamically linked versions of

• SQLite Encryption Extension
• wxSQLite

on top on original SQLite library? This can help to fix https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=906527
BTW, can you build encryption extension without SQLite patching? There is problem mentioned in link above:

SQLCipher is a customized version of SQLite3 that adds encryption support.
Unfortunately, this cannot be implemented as a plain plugin to SQLite
because it is not possible to make the encryption transparent without
modifying the core of SQLite.

[utelle]

Will the SQLite Encryption Extension separation allow to build dynamically linked versions of

• SQLite Encryption Extension
• wxSQLite

on top on original SQLite library?

Unfortunately, the answer is no. All encryption extensions, even the official one, SEE, need access to certain internal SQLite functions to support database encryption, and the wxSQLite3 encryption extension is no exception. Those functions are not part of the external SQLite API, and are therefore not accessible, if you link dynamically to the original SQLite library.

This can help to fix https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=906527

Starting with wxSQLite3 version 3.5.0 (released in January 2017) the SQLite source code is an integrated part of wxSQLite3. Obviously, the Debian package team stopped to upgrade their wxSQLite3 package after version 3.4.1. The above mentioned issue can't be fixed without taking the decision to allow the wxSQLite3 package to use its own adapted SQLite version including encryption support.

BTW, can you build encryption extension without SQLite patching?

Well, in fact wxSQLite3 does not apply any patches to the original SQLite source code. It simply uses some additional C preprocessor instructions on compiling the code. That's all - no more, no less.

There is problem mentioned in link above:

SQLCipher is a customized version of SQLite3 that adds encryption support.
Unfortunately, this cannot be implemented as a plain plugin to SQLite
because it is not possible to make the encryption transparent without
modifying the core of SQLite.

AFAICT Debian supports only an outdated version of SQLCipher. However, they allowed for the SQLCipher package, what they didn't allow for the wxSQLite3 package for unknown reasons: namely including the source code of SQLite itself.

Up to now SQLCipher needs to actually modify the SQLite source code to implement certain extensions like customized pragma commands. Unlike SQLCipher, wxSQLite3 does not modify the SQLite source code. However, as explained above, wxSQLite3 still needs to be able to use internal SQLite functions. Therefore the "problem" can't be solved.

Current versions of wxSQLite3 allow to build a separate SQLite library with encryption support which does not depend on wxWidgets. Build files for this purpose are included since version 4. However, several developers asked for a separate package not including the wxWidgets wrapper. To fulfill this wish is the goal of the SQLite Encryption Extension separation.

Unfortunately, I had to postpone this separation process for several months due to personal matters. However, maybe that wasn't that bad after all, because there were changes to the public SQLite code on Feb 7, 2020: “Simplify the code by removing the unsupported and undocumented SQLITE_HAS_CODEC compile-time option”. AFAICT, these changes will take effect with the release of SQLite version 3.32.0. As a consequence, all SQLite encryption extensions out there will not be able to easily support SQLite version 3.32.0 and later.

In late February I started to work on a new implementation of a SQLite encryption extension that will be able to support SQLite 3.32.0 and later. However, as before it will be required to compile SQLite itself and the encryption extension together, because even this new implementation depends on certain internal SQLite functions.

If all goes as planned, I intend to make a first preliminary version of the new implementation publicly available within April for testing by volunteers.

[utelle]

A first preliminary version of the new implementation of the encryption extension is now available in project SQLite3MultipleCiphers.

Feedback is welcome, and help in testing and discussing further development will be highly appreciated.

[Willena]

Hi @utelle,
Thanks for the fantastic work. I can now remove (archive) the repository that I kept updated until now for the SQLite 3 JDBC Connector.
I'll try to integrate the "new" library in the JDBC connector soon. I'll give you feedback if needed.

[Adriatic1]

Any plans to support future sqlite 3.32.0 which has removed support for SQLITE_HAS_CODEC compile time flag ?

[Adriatic1]

Ignore the question, just checked https://github.com/utelle/SQLite3MultipleCiphers :)
Thank you for your great work!

[utelle]

According to the SQLite website (draft) the release of SQLite 3.32.0 is planned for end of May 2020. I'm working on getting my new project SQLite3MultipleCiphers ready to replace the current implementation of the encryption extension in wxSQLite3 thereafter.

However, it would be really good to get feedback from other developers whether SQLite3MultipleCiphers works for them or whether there are glitches that have to be fixed.

[ga2k]

Hi, I use wxSQLite3 extensively. Everything I write is a DB, and your library is wonderful.

I like the namespace idea. A simple search and replace will fix all my legacy code.

Something I would like to see, administratively, would be for you to submit your code to vcpkg for downloading and installing via that mechanism. That package manager makes life so easy for moving to new machines, and so on. It also reduces the complexity of my repositories, as I don't have to set up externals to load your code where the compiler can find it. Now, I'm off to the mailio git to ask him to do the same thing,

Cheers,
Geoff.