Opencast full rewrite in 2025?

[LukasKalbertodt]

This was still posted on the mailing list (without replies yet). I'm just copying the text here for migration.

---

Sup,

I sure know how to get your attention with a subject line 🙂

In a meeting yesterday, the Spring Security update came up briefly. And it got me thinking: is it actually time to rewrite?

I know, rewriting Opencast is a meme. One that has been around for longer than I've been part of this community. But let's actually have a proper conversation about this for once. Where everyone can bring their arguments, experiences and assessments to the table in order for us to make an informed decision at the end.

But wait! Despite what most of you were probably suspecting, I am not bringing this up to replace Java. This is truly not my point! So let's please completely set aside the question of programming language, to narrow the scope of this discussion.

Also note: while I'm leaning toward "yes, rewrite" right now, I don't want to convince you – I merely want to start a discussion and hear from all of you.

With that meta stuff out of the way, let me lay out my view of the situation.

---

Reasons for

Coming back to Spring Security: the outdated version has been a problem for many years and no one managed to update it yet, not even Greg! The amount of time sunk into maintenance work like this is enormous. The state of the codebase significantly slows down the implementation of new features and other changes. Opencast often requires arcane knowledge to work with and code in: I have lost count of how many times, in a meeting, a part of Opencast was mentioned, only for many experienced devs to chuckle and say "no one understands that, I wouldn't touch it with a ten foot pole".

Random crashes, unexplained behavior and inconsistent data are common. There is so much unneeded complexity, so many code smells, edge cases and bugs. I cannot remember when I last tried an Opencast feature that works flawlessly, or when I last looked at Opencast code that looked reasonably fine to me.

As you know, Opencast recently received a large fund in order to modernize, refactor and generally improve it. We had a couple of meetings about this and agreed on some good improvements already, but the coding work has not been started yet.

I am more and more convinced that the "incremental improvements" route is actually slower than a full rewrite. That's the main reason I'm writing this mail. Yes, rewrites take a ton of time, but the velocity is way faster. Imagining trying to implement the discussed "data model" changes gives me shivers. I don't see how this can be merged in less than half a year, and this is just a first step.

A fresh start means we can learn from our mistakes, get rid of unnecessary complexity, part with historical baggage, and build a more robust and maintainable product in the end. This funding is the perfect opportunity: if not now, it will never happen.

Learning from mistakes and backward-compatibility

Of course, this requires us to actually learn from our mistakes. Just starting a rewrite aimlessly will get us nowhere. We need to really understand the pain points of devs, admins and users of Opencast. And find out what ultimately causes them and how we can improve. We don't want to just rewrite the code, but also improve on Opencast's data model and semantics.

But we cannot throw out everything either. We want current users of Opencast to eventually switch to the rewritten version, so a certain amount of backwards compatibility is required. We probably should keep the ingest API for capture agents, for example. This requires careful consideration, but in the end, there needs to be a reasonable migration path for admins of installations and for developers of third-party integrations!

Reasons against

My biggest worry is: I am still too naive and just wrong about a rewrite being faster than incremental improvements. This is what I'm most interested in hearing your assessment of and opinion on!

Even if the rewrite is faster, it will take a while, during which we have two parallel codebases. That means some feature might need to be developed twice and/or people are discouraged from putting any work into the old codebase since it will be thrown away at some point. We had a similar problem with the admin UI before.

I also want to touch on a topic not commonly talked about: a rewrite means that experienced members of the community have to re-learn several aspects and some of their knowledge will become useless. It's easy to dismiss this as irrational, but Opencast is worthless without its community and disregarding "human problems" like that doesn't get us anywhere. I personally think that a rewrite is possible without alienating or splintering parts of the community, but I understand if others feel differently about this.

Finally, a rewrite is scary: if we run out of money before reaching a usable state, we have essentially nothing. With incremental improvements, we can stop at "any time" and have still gained something. We'd be leaving the safe and familiar shores.

---

Thank you for taking the time to read this. I am truly curious to hear what you think.

Have a nice Christmas and see you in 2025!

Lukas