Atomic Bomberman is lagging #14
Comments
|
Hello, As you can probably tell from looking at the issues list on here, I've been struggling with some people encountering lag or general weirdness but never managed to pin down the cause. In your testing with VMWare, are both VMs running on the same host, on the same network or across the Internet?
What are you using to simulate packet loss? Thanks |
|
In your testing with VMWare, are both VMs running on the same host, on the same network or across the Internet?
Both VMs running on same host.
What are you using to simulate packet loss?
VMware lets you to set fake packet loss, ping or bandwidth in network adapter settings.
|
|
So, the logs you attached were from an actual play session, not the VMWare packet loss test, right? They're suggesting packet loss may be going on (1player transmitted 3,182 packets vs. 2player receiving 1,242 packets), but some of that could be down to them not quite lining up time wise (i.e. one game being exited before the other). |
|
Yes, those logs from actual play session, but when recording those logs we didn't wait for actual gameplay because the synchronization (Receiving level data) taken more than ~20 minutes with debug messages enabled, so we closed the game at almost same time. Player 2 was host in this session, do you mean that player 2 fails to receive packets? |
|
Player 1 sent 3,182 packets and Player 2 received 1,242 |
|
Tested in VM (without packet loss and etc) and got those results: Something clearly wrong in real game session Any idea where to dig? |
|
I spent a bit of time fiddling earlier. Bomberman is sending lots of small broadcast packets which seem to come through okay with the native IPX driver on XP, swapping in IPXWrapper degrades performance slightly, but no obvious cause. Switching to DOSBox encapsulation degrades it badly. Sending lots of UDP packets like this isn't a "done" thing, and as a result the IP stack of any involved machine could be dropping, throttling or awkwardly shaping the traffic. Depending on how bomberman processes traffic, I might be able to introduce a feature to coalesce frequent-but-small packets into fewer batched ones, which might help. First though, I'm finishing off a profiling feature I was working off before which will hopefully clear things up. |
We can't test if IPXWrapper UDP/real IPX performs significantly better right now, but will post results here when possible. This is DOSBox IPX tunneling issue or with current implementation in IPXWrapper? Never had issues with it in DOS games.
Is just adding some padding to make packets weight more may solve issue? This looks easier to implement. Or this not how it work?
Do your best. It's great that projects like this exist. |
|
The issue isn't the packets themselves being small, but the sheer number of them. Bomberman is sending 400+ (broadcast) packets every second when in a network game (most of them <50 bytes), which is resulting in it spending a lot of time sending/receiving packets. I don't have profiling data to back this up, but I can only assume the original IPX driver in Windows was a lot more simplistic than the IP stack in modern Windows. It seems to be spending around 1s inside the IPXWrapper I experimented a bit with using I'll have to try and find more optimisations that can be made elsewhere. |
This was an experiment to see if using overlapped WSASendTo() would improve performance over vanilla sendto() when sending an obscene number of tiny packets like Atomic Bomberman does (#14). Turns out this actually makes performance *worse*. The initial WSASendTo() calls take even longer than the sendto() calls did, not counting any time spent afterwards dispatching the completion callback and cleaning up. Back to the drawing board.
|
So there are still places within IPXWrapper where I might be able to squeeze a bit more performance out where certain APIs are being used in terrible ways by Atomic Bomberman, but I still think the root cause here is the sheer number of packets. I've attached a prototype build of IPXWrapper which batches and merges packets when the application starts sending them at ridiculous rates (~90% packet count reduction during AB gameplay). Can you and your friend give it a try and report how it works? There is now a profiling option in ipxconfig you can turn on to gather useful statistics for this which shouldn't slow the game down like the full debug log does. |
|
It's a little better now, but still players still move jittery (but smoother than before) and bombs still may do not appear on other side in time. Synchronizing at beginning of match are faster now too Prof Player 1.log While recording log with debugging enabled were able to reach gameplay, but players doesn't move at all. (While profiling enabled everything is "fine") |
|
Well that raises more questions than it answers. Going by the Prof logs there, the packet coalescing I implemented isn't actually kicking in (there should be a message along the lines of "High send rate to xxx detected, coalescing future packets"), so nothing else in my build should be significantly faster. The stats in the log back this up too: If the packet coalescing was working, the UDP socket stats would show far fewer packets than the IPX socket stats. The counters suggest there is a lot of packet loss going on, which isn't too surprising at the packet rates above. |
|
Placebo effect in action ¯\_(ツ)_/¯
On Monday, November 27, 2023 20:37:04 (+05:00), Daniel Collins wrote:
Well that raises more questions than it answers.
Going by the Prof logs there, the packet coalescing I implemented isn't actually kicking in (there should be a message along the lines of "High send rate to xxx detected, coalescing future packets"), so nothing else in my build should be significantly faster.
The stats in the log back this up too:
[988646.95, thread 5088] IPX sockets sent 31939 packets (1089730 bytes)
[988646.98, thread 5088] IPX sockets received 3009 packets (104774 bytes)
[988647.03, thread 5088] UDP sockets sent 31939 packets (2047900 bytes)
[988647.06, thread 5088] UDP sockets received 3009 packets (195044 bytes)
If the packet coalescing was working, the UDP socket stats would show far fewer packets than the IPX socket stats. The counters suggest there is a lot of packet loss going on, which isn't too surprising at the packet rates above.
—
Reply to this email directly, view it on GitHub <#14 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIU6SQKV3JAS7S6CVHD2G4LYGSXSBAVCNFSM6AAAAAA7RQVG6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRYGA3TCNBVGQ> .
You are receiving this because you authored the thread.
|
|
It really slightly better now, but not due to changes in wrapper.
On Monday, November 27, 2023 20:37:04 (+05:00), Daniel Collins wrote:
Well that raises more questions than it answers.
Going by the Prof logs there, the packet coalescing I implemented isn't actually kicking in (there should be a message along the lines of "High send rate to xxx detected, coalescing future packets"), so nothing else in my build should be significantly faster.
The stats in the log back this up too:
[988646.95, thread 5088] IPX sockets sent 31939 packets (1089730 bytes)
[988646.98, thread 5088] IPX sockets received 3009 packets (104774 bytes)
[988647.03, thread 5088] UDP sockets sent 31939 packets (2047900 bytes)
[988647.06, thread 5088] UDP sockets received 3009 packets (195044 bytes)
If the packet coalescing was working, the UDP socket stats would show far fewer packets than the IPX socket stats. The counters suggest there is a lot of packet loss going on, which isn't too surprising at the packet rates above.
—
Reply to this email directly, view it on GitHub <#14 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIU6SQKV3JAS7S6CVHD2G4LYGSXSBAVCNFSM6AAAAAA7RQVG6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRYGA3TCNBVGQ> .
You are receiving this because you authored the thread.
|
|
I haven't been able to come up with any reasons why the packet coalescing might not kick in, so could you try this build out and send me the logs with profiling enabled? Hopefully something in there will point at why it isn't working. Thanks |
|
It's definitely working now and makes game playable, but there's still issues: Perf Player 1.log |
|
Unfortunately, the Atomic Bomberman network code just isn't very good. The best I can do is tweak IPXWrapper to reduce the chances of packets getting lost and improve the speed of the calls it is (over)using - as you've seen, when packets are lost it can go out of sync and makes no attempt at correcting the state between players. The walking animation not playing isn't even specific to IPXWrapper - that happens when using the native Windows IPX driver on XP too. I've made a few more optimisations in this build which might help a little bit, let me know how you get on. |
I understand that issues may appear on software/hardware game designed for. We didn't noticed any difference with this build |
|
We also tested game with 10 bots and it's looks unplayable if there's more than 3 players (for obvious reason) |
|
Hi. I was testing Atomic Bomberman with Ceotom a month or two ago. |
|
@insellium I've not got any further leads/improvements to follow at the moment, so no progress on the Atomic Bomberman front. |
|
We had a LAN party with 10 players using 5 computers. The game got async using windows and the ipxwrapper version linked in this issue. We switched to Fedora + Wine + same wrapper version and the game was super stable and without lags. |
We have 2 players with following setup:
1 player: Hosts DOSBox ipxnet server (using DOSBox-X)
2 player: Connects to 1 player's ipxnet server
Both running Win10, who hosts in-game doesn't matter. Players teleports continuously and doesn't properly sync some events, like bomb placement, power-up pickups. Bombs can appear at wrong places or too late, power-ups may re-appear at same place after being picked up and etc.
Video from 1 player side (red in-game):
https://github.com/solemnwarning/ipxwrapper/assets/36301121/816715b9-57d1-4ac1-a1c6-39fa43489b80
You can notice how 2 player walks jerky
17s: 1 player pickups disease that forces player to set bombs continuously as soon as possible, but at 19s it reappears below bomb
25s: For some reason 2 bomb unexpectedly appears behind 1 player which is impossible normally (he can set only 1 bomb)
Logs with debugging messages (we didn't wait for the game to start because it takes VERY LONG to start with debugging enabled and we quit on "Receiving level data"):
1player.log
2player.log
During test with VMware VMs with this setup:
Host machine: Hosts DOSBox ipxnet server on Win10
VM1 and 2: Connects to host's ipxnet server using host-only adapter, running WinXP due to performance reasons, 2 VMs with Win10 are too slow.
I'm found that those issues appears only if simulate packet loss, ping just add delays until you action appear on other side.
Pinging showed that we has ~38ms ping and 0% packet loss. Looks like ipxwrapper drops packets?
The text was updated successfully, but these errors were encountered: