Add a docker-compose.yml

[davepeck]

This allows developers to spin up a running Emissary instance with docker-compose up.

It's not yet mergable, however.

The main issue has to do with the setup/config step vs. the actual running of the server:

When a dev sets up an Emissary server with go run server.go --setup, a config.json is created, and a number of entries are added to mongodb — particularly when creating new domains.

We need both of these to be available when the server starts up. To manage this, I configured once and have special config-docker.json and dev-db.tar.gz (a mongodb dump) files checked in. The emisarry server container uses config-docker.json; the mongodb container's entrypoint first restores the dev-db.tar.gz and then spins up mongo.

Not very pretty and probably not quite what we want, but figured it was a place to start the conversation.

(A second issue: Domains assume an SMTP server; you can't log in the first time without getting a reset link. I created a simple SMTP logger service in tools/smtp/smtpLogger.go that echos out whatever email is sent to the console; this is run as the smtp service in the docker-compose.yml. It works and it's handy but, again, maybe not our ideal.)

Hopefully this is a useful way to start discussion about what we really want. And feel free to close/no-merge this if it's not!

[benpate]

This is fantastic. Thank you!

RE: Emails on startup. Yes, this is probably not optimal. It was built with production systems in mind -- where admins shouldn't necessarily have users' passwords -- but it is a pain for standing up a development system. I've used MailTrap for this, but I can see how it might be best to just ditch the SMTP requirement. I'm certainly open to suggestions from someone looking at this with fresh eyes.

RE: Mongo Config. I've considered having a special --demo mode that auto-configs a single domain (like localhost, or similar) so that you can get up and running super fast. Perhaps this might work well with a "demo" Docker instance.

I think I've seen something like this in other projects -- is it pretty common to have different Docker distributions for "demo" vs. "production"?

Again, this contribution is much appreciated. Thank you!

[davepeck]

is it pretty common to have different Docker distributions for "demo" vs. "production"?

Yes, it is. And other variations on those themes. (You'd like them all to share as much in common as possible, of course.)

Your question gets to the center of it: what do we want docker for to begin with?

• Is it so that a developer can use a single command to stand up all the necessary services and make code changes/submit PRs?
• Is it so that a potential user can get a feel for Emissary before deploying in production?
• Is it to help define what the production infrastructure should actually look like?

Maybe, over time, all of these. But where to start? Maybe just making it easier to kick the tires in a low-commitment way?

Related, I'm a little unclear about how much you can do with Emissary without tying it to a real domain. Curious how you think about this.

(Also: I have a different approach to docker that I think I like better as a baseline on this branch here. Basically, it stands up both a "setup" and a "real" service, at ports 8888 and 8080 respectively, and has them share the same config.json and db. There's no 'dev-db' or pre-canned config. Thinking of "setup" as a separate service sorta feels right, given the current design, and removes a lot of cruft that's in this current PR.)

[benpate]

Yes, I think I agree with where you're headed on this. Hopefully I'm addressing your points here (apologies if I miss one)

Docker
For where we are now, my first thought is that Docker should be an easy way for a new developer to stand up the server and kick the tires. Other use cases may be more important in the future, but that seems like the first place to bootstrap this software community.

On that thread, I'm going to spend this morning on a couple of things to make the initial setup a little nicer on localhost -- especially around emailing user passwords.

Setup Service
Yes, I agreee that --setup should be a separate service. I tried to isolate it to protect production instances from hacking, but this is another thing that is a little clunky for first-time devs. So I love the idea of the docker network starting a setup server automatically.

What's Possible on localhost
Also, I know I need a TON of "getting started" content -- probably videos that will help with the "what can I do?" question. One thing I didn't manage to squeeze into the presentation is that RSS+ActivityStreams polling is always a fallback to ActivityPub. So you can always follow a remote instance from localhost, even if you can't get ActivityPub notifications.

What I've done (and what ran in the demo) is to have two local servers: localhost and 127.0.0.1. On my machine I actually have three or four local domains running at once. Let me know if this would be a good topic for a first explainer video and I'll try to shoot one this weekend.

[davepeck]

Docker should be an easy way for a new developer to stand up the server and kick the tires.

👍

especially around emailing user passwords

One thing I noticed is that the HTTP port is not included in the emailed password reset link.

Related: is the expectation at the moment that all domains use the same HTTP port?

Yes, I agreee that --setup should be a separate service.

Yeah this feels good to me. I've made this change.

Now when you run docker-compose up you get the --setup service at :8888 and the running app at :8080.

Right now you've got to manually restart emissary when the config.json changes (docker-compose restart emissary). I'll wrap the emissary service in a file watcher next.

What I've done (and what ran in the demo) is to have two local servers: localhost and 127.0.0.1.

This brings up another question: can a single running instance of emissary handle multiple domains at once? AKA if I use localtest.me, set up and then visit http://vanguard.localtest.me:8080 and http://catalyst.localtest.me:8080, should things work fine?

After this, having a single button in the emissary setup UI that lets you "configure to kick the tires" would be really fun -- maybe set up a couple localtest domains, go through that intro content you're thinking about making, and you're off to the races?

[benpate]

I probably overlooked the HTTP port in the outbound email. I'm adding a ticket to have that included. I probably expected that any production site would just be the default port, but this would help for development, for sure.

With this new, easy way to run --setup and the main server side-by-side, you're going to make me actually like using Docker. :)

About config changes - Emissary uses fsnotify to watch for changes in the filesystem and should automatically update when there are changes. If that's not happening for you, my first guess is that Docker may be getting in the way, somehow. Does Docker make a virtual filesystem? Or a better question: is there an easy way to listen for filesystem changes in a Docker container?

Multiple Domains - YES. A single Emissary server will server multiple domains -- each from their own dedicated Mongo database. All domains would listen to the same ports, and the server routes requests using the Host header. You should be able to set up a second (and third, and fourth) domain in the --setup tool - just click on "Domains" in the navigation bar and add a new one. So, your "localtest.me" examples should both work without a problem.

Since we now have a Dockerfile to work with (I'm eternally grateful to you for this) I'm thinking the best way to do a "demo mode" would be to set up one or two domains in the Docker network and ship with a default configuration that points to them. Since you can specify the config file on startup, we just ship with a separate config file called "config-docker-demo.json" that expects to run in the Docker demo container.

Does this make sense? I'm happy to make the config file if you don't want to - either way it should be pretty easy - once you're comfortable with the Docker container.

[davepeck]

Emissary uses fsnotify to watch for changes in the filesystem and should automatically update when there are changes.

Ah yes, this does appear to be working. Excellent.

I think I assumed I'd see some sort of "config reloaded" message in the logs.

Then, after I get a few small clean-ups in, I think we're in a pretty good place here? I'm happy to update your docs if you like, too.

set up one or two domains in the Docker network

Just to be clear, most docker tooling doesn't configure your host system's DNS settings (for instance, Docker Desktop for Mac doesn't) -- one reason i resort to stuff like localtest.me in these cases

[benpate]

There (should be?) a log message -- generated with zerolog -- but it may not be showing if you don't have logging turned on. In the setup tool, look at Server Settings > Testing and Development. Set the "Debug Output" level to "Trace" and you'll see everything.

Sorry so little of this is documented. The logging levels are relatively new, and I haven't added that into the quickstart :(

Thanks for the info about Docker's networking. I've read up on it a bit, and I thought it maintained internal named, or something like that, but it makes sense that it wouldn't change the host's DNS. I'm a complete newbie when it comes to this, so I'll throw myself at your mercy -- whatever you think is best.

One more thing: I'm nearly done with an update that will allow server admins to set and update passwords for domain owners -- ON LOCAL DOMAINS ONLY. It also makes SMTP configuration optional. This seems like a decent compromise that maintains some security for production sites, while still making it simpler for developers to kick the tires. I'm thinking it'll be done tomorrow, and I'll get it up on the main branch for you all to play with.

[davepeck]

I thought it maintained internal named

Yeah, each service in a docker-compose.yml gets a name that other services can use to reach it (aka the emissary service can reach mongo at mongodb). But these aren't exposed on the host machine.

I'm nearly done with an update that will allow server admins to set and update passwords for domain owners -- ON LOCAL DOMAINS ONLY. It also makes SMTP configuration optional.

This seems great. I figure we don't want to keep that smtpLogger stuff around if we can possibly avoid it!

(Getting us halfway there: I've removed smtpLogger.go and Dockerfile-smtpLogger in favor of directly referencing ghcr.io/literalgarage/smtp-logger:latest in our docker-compose.yml. That's a docker image I maintain. I assume we'll remove the smtp service entirely from docker-compose.yml when you've got the updates to support it.)

[benpate]

Hey Dave ~~ thanks again for building this. I held off on merging it because you said you weren't quite pleased with it yet. Do you think this is "done"? I'm happy to try it out to see that it works for me (then merge) once you say you're pleased with this code.

[andypiper]

I'm eager to try this out on my home Synology where I have Docker running, also waiting to hear more about the shape this is deemed to be in! 🙂

[davepeck]

Hey Dave ~~ thanks again for building this. I held off on merging it because you said you weren't quite pleased with it yet. Do you think this is "done"?

Yes; I think it's done and ready to be merged at this point.

The one thing I wasn't sure about was the SMTP logger that the docker-compose spins up. I think it works well, but I didn't know if you were doing work on your end to remove the need for it entirely.

All this is to say: if you're happy with the state of the PR, I am... merge away. :-)

[benpate]

Merged.. lol yolo :)

[benpate]

Hey @davepeck - I've dig into the work you did. Thanks again so very much! I have made some changes to the application and the docker configurations (in the dev branch) based on several of the issues you raised with this. Two important things:

1. I've made separate docker configurations for "demo" and "production". I still need to test that the production config works, but I think it will help me a lot with my hosting at Digital Ocean. Specifically, it auto-installs FFmpeg, which will be important to the platform with the next set up updates.

2. I've added a default config.json file for the "demo" setup, and a new features to automatically insert an admin/admin user on initial startup. This makes "kicking the tires" as simple as running docker compose, then opening your web browser. Hopefully you don't mind me refactoring some of your code as I dug in to figure out how all of this works. :)

I'll get this documented on the website soon, and ready to release to the main branch. I think this will be a huge step forward for the whole platform. Thank you, thank you, thank you!

[andypiper]

This is great!

I've yet to be able to try it fully. Mongo is my sticking point. Both on my Synology, and on my Pi-based K3s cluster, Mongo 5.x will not start as the image needs something fancy (AVX CPU extensions, apparently). Definitely not an Emissary thing though, so I shall continue my quest to find a place to play with this. May be worth noting, however, in case others try the same things I have!

[benpate]

Sorry to hear that, @andypiper

I have a Synology, but don't use it for anything more than file backups, so I can't be much help there. Docker does work pretty well on my Mac.. I might start trying to do more things with it there.

I've been told that FerretDB is a good OSS substitute for Mongo. I may experiment with it to see if it's a drop-in replacement, and will report back when I have any news.

One other alternative is that the MongoDB doesn't have to be on the same machine. I've run an early version of Emissary using a database hosted in the cloud -- I think MongoDB Atlas has a free/trial tier that could work for you.

[davepeck]

Huh, interesting — bummer that Mongo's docker image doesn't run out of the box on a Synology; you'd hope that would Just Work™. And yeah, you can definitely edit the docker compose, comment out the mongodb stuff, and point Emissary in the right direction.

PS Andy: I realize I should have written documentation for this before we merged. But the basics are easy: docker-compose up and then visit :8888 to configure your Emissary server. Once configured, visit :8080 to see it in action.