Migrating away from acid-state

Hi there,

I've got some open source budget at work, and would like to put it towards hackage-server. In particular, my thoughts were to replace the usage of acid-state with a postgres database. By my reading, this would allow us to:

use traditional database backup techniques
remove a huge amount of code around managing tarball backups
dramatically lower hackage-server's memory footprint
probably solve downloads counts stored inefficiently. #1276
remove the custom search engine implementation
allow for migrating data separately from shipping new binaries, which in turn will likely improve uptime

All of this is likely to help #1467 as well.

I just wanted to ensure there is an appetite for this work before I invest too much time. Happy to setup a call and discuss the finer details as well.

There's #1465 so you probably want to talk to the author of that PR.

Yes, we're very open to this change, and coordination with the two authors of #1465 would be the way to go

Hi author of #1465 here 👋 .

So far the PR has received feedback on how to use beam, which are valuable but there are still some open question in the OC of the PR regarding the right strategy to make progress in a way it's mergeable. Maybe this issue could kick some discussions.

I personally feel it would be more encouraging to allow smaller PRs to be merged while the whole feature is developed. That way we avoid working for longer time in a branch. This would allow us to focus on how to structure the database feature and having migration without how it is actually used. Porting each feature to DB could be handled independently. Definitely some exploration to know if we are in the right way is needed in advance. That was my idea of the current PR.

The PR didn't cover specs which I think is a must.

When a feature is ported so far the idea was to port acid to db on boot and then read/write to the db only. I think that is reasonable since we can't guarantee acid and db to be in sync and implement writes on both backend. Yet, to allow smaller PRs we could have a feature flag to choose when to migrate to DB backend.

If we take that approach it means the feature code will need to handle both backends, yet only one is going to be used. It adds complexity, but writing to the acid state is still needed if we want to write tests probably and we could structure specs to check read operations behave the same.

Another open question regarding specific features is what is the desired db schema. Should each feature have it's own table or should we embrace a more common "users" table where each feature will write certain tables?

My preference to make progress would be

Have a PR with database feature with a defined migration solution for sqlite and pg
Have another PR with a small feature migrated to db with a feature flag and spec to be used a reference

But some endorsement or feedback from maintainers would be useful IMO.

In #1465 I started with users table to familiarize myself with the code and as I thought it was better to do manual testing as I needed to create users and familiarize with the user registration for local testing either way.

Maybe download counts is a simpler feature to start with. But download counts is a broken feature (see https://discourse.haskell.org/t/hackage-statistics/12919, #458). So it will not be really impactful. Worth porting if it's going to be fixed somehow eventually.

My plan is to:

Construct an idiomatic, normalized set of table schemas (not 1:1 with the acid types)
Write a new executable to export the acid state as SQL insert statements
Split the makeAcidic calls/logic into their own modules
Move the new Acid modules into the export executable
Fix all the compilation errors by writing new sql queries

Points 3 and 4 are desirable in that they offer a very clean separation---keep the acid logic only in the exporter, excising it completely from the main lib. Furthermore, they are small, easy-to-review PRs. In addition, having all of this code localized into a single cabal component means it can easily be deleted wholesale sometime in the future.

Once I have enough of the work done to ensure I haven't gone down any garden paths, I will begin opening PRs.

While I hear the argument to do this piecemeal and ideally feature-gated behind runtime flags, I don't think it's technically feasible. I have somewhere between three and seven weeks of contiguous, full-time engineering effort to throw at this problem, and a refactor of this scale is no small task; doing it incrementally is significantly more engineering and political expenditure than I have to spend.

This seems to me to be firmly in the "done is better than perfect" category, and I intend to push forward with that mindset unless I hear explicitly against it from the maintainers.

I recall in AmeriHack that migrating on boot was desired instead of having a separate executable.

But I think is cleaner to move all the make-acid to its own module.

I wasn't very familiar with it so I envisioned using the feature api to reach the state to migrate. As opposed of using acid directly. This is treating acid more as a black box. But both approaches work and your moves the code closer to the final form I think.

Next couple of weeks are work intensive for me, but poke or share things if you think I can be useful somehow.

PG and Sqlite are wanted as a backend go avoid requiring PG for smaller instances.

I don't think piecemeal is necessary in either approach, and they aren't so different to be honest. Having one executable vs two is not the hardest thing. The tricky part is just rewriting the logic that presumes there's acid-state available vs moving to a database approach. Certain things will just need to be done very differently.

PG and Sqlite are wanted as a backend go avoid requiring PG for smaller instances.

Personally, I don't understand why any on-their-own-time contributor would care about non hackage.haskell.org use of hackage-server. I find the master / central-server branch split causing more problems (if nothing else it's confusing) than providing benefits. All other than hackage.haskell.org repositories I'm aware of are not using hackage-server (but just listing tarballs and creating index with other tools, e.g. head.hackage.haskell.org or https://github.com/IntersectMBO/cardano-haskell-packages)

Having to deal with both PG and SQlite may be worse than dealing with just acid-state.

@phadej

@phadej uh oh. This is the first I've heard about central-server. Given that my goal is to improve hackage.haskell.org, which branch should I be working off of?

I am in agreement re sqlite; setting up postgres is not onerous, and having one fewer codepath seems very desirable to me.

@isovector

@isovector I think master if your changes are generic, and central-server if they are not. AFAICT master is merged into central-server once in a while. Having two long lived but somewhat separate branches is not great. And there are small things (like central-server not having CI setup) makes it even less great.

One could even argue that getting rid of central-server (i.e. making whatever changes needed to hackage.haskell.org be configuration, not code changes) would be even more important for long term health of this project.

@isovector

@isovector please ignore central-server for this PR. its irrelevant and oleg's comments on it are irrelevant.

on pg and sqlite the idea is its not hard to support both and its not only an issue of running things but easy local development and CI etc. being able to develop and test against sqlite lowers the burden for contributors.

If CI tests are not run against PG, are they testing all the right things? To me "easy CI" (and local development) is lousy argument, especially as having postgres server on GitHub actions is one of the easiest services to setup.

I have no stakes, so indeed my comments can be ignored.

I also have no stakes, so take my comments or leave them, but I agree that:

tests should be against PG
setting up PG in CI is actually pretty easy

and moreover I think that setting up PG for local dev isn't very hard either, or at least doesn't have to be if we provide scripting support. Have a script that launches a PG instance specifically for local dev, with all its data and configs in some local temporary directory. Doesn't have to be a heavyweight thing.

@isovector

@isovector If you're gonna use rel8, is there any way to test that the query plans are using the right indices? Does anyone know if circuithub does any testing like this? I am worried because I know the generated SQL can get real esoteric.

@ysangkok

@ysangkok I was planning on using rel8, since the developer experience is significantly nicer than any of the alternatives. I'm not aware of any direct approaches to test index usage, but am not particularly perturbed by this for a few reasons:

all of our queries (so far) are very simple
rel8 runs on top of hasql, so we can always swap in raw sql for any queries which happen to get too esoteric

If the concern is merely from an auditing perspective, I can think of a few ways we might accomplish this:

log some % of executed queries & analyze them offline and after the fact
write all queries as top-level declarations, which would enable us to implement something like allProperties, whose execution explanations we could then iterate and check programmatically.

Let me know what you think.

I don't think declaring all queries as top level is necessary at first, but it's good to have that option available, I suppose. If we start out with just logging/monitoring, then server administrators can run EXPLAIN ANALYZE manually for any problematic queries. But, I am not sure how monitoring should exactly work, and I don't know whether existing admins have any infrastructure in place for that.

I also have no stakes, so take my comments or leave them, but I agree that:

tests should be against PG

setting up PG in CI is actually pretty easy

and moreover I think that setting up PG for local dev isn't very hard either

I have been using testcontainers, which has a Haskell implementation that I maintain. I use it to test beam-postgres itself. I think that forcing Hackage to use Postgres, both for local development and in production, is a lot more ergonomic than it used to be!

Navigation Menu

Repository navigation

Migrating away from acid-state #1486

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Migrating away from acid-state #1486

Description

Activity

phadej commented on Apr 10, 2026

gbaz commented on Apr 10, 2026

bcardiff commented on Apr 12, 2026

isovector commented on Apr 13, 2026

bcardiff commented on Apr 13, 2026

gbaz commented on Apr 13, 2026

phadej commented on Apr 15, 2026

isovector commented on Apr 15, 2026

phadej commented on Apr 15, 2026

gbaz commented on Apr 15, 2026

gbaz commented on Apr 15, 2026

phadej commented on Apr 15, 2026

bmillwood commented on Apr 26, 2026

ysangkok commented on Apr 26, 2026

isovector commented on Apr 27, 2026

ysangkok commented on Apr 27, 2026

LaurentRDC commented on May 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions