Are We Fast Yet Benchmarks #225
Comments
|
The overall performance, comparing the interpreter-only performance of SOM implementations looks like this:
ykSOM isn't bad :) |
|
Which of the SOMs do the "treat |
|
PySOM and TruffleSOM do various optimizations, including inlining.
The AST interpreters are also more aggressively inlining loop constructs like SOM++ does only a broken inlining of ifTrue/ifFalse. CSOM doesn't do anything. And SOM-RS doesn't really optimize either. |
|
When you say "inline", my memory of SOM++ is that it hardcodes Basically, I'm trying to work out what the comparison is :) From memory, the SOM++ |
|
PySOM and TruffleSOM implement inlining of the blocks associated with these operations. This means, if you change True.som and False.som to do extra things in the Have a look at
I tried to be good and document some of these bits:
Unfortunately, I didn't link the ReBench reports back then. |
|
Can I suggest that none of the SOMs should do the "standard Smalltalk cheating" and all SOMs should do only transparent optimisations? I think that it's very hard (and, arguably, unfair) to compare the performance VMs doing transparent optimisations (including, by the sounds of it, PySOM and TruffleSOM!) vs. those that are not (e.g. SOM++). |
|
Well... I am convinced enough that all these optimizations can be made fully transparent with only minimal startup cost. But, I haven't put in the engineering to actually do it... So... yeah. Though, I am not sure what's unfair about comparing VMs that do and that don't do optimizations. How else would you know which optimizations are beneficial? |
|
When I say "transparent" optimisation I mean "any change the user makes to any FWIW, I think a fully transparent |
|
To make it transparent I was thinking of possibly doing one of the following:
If the check fails, disable, and possibly warn the user. I wouldn't go so far as to make it fully generic... |
|
I think that even with these rules, users can't override |
|
Override is just a cheap guard and a resend away. I have to admit it's not implemented though... (Strictly speaking that's not entirely true. It's implemented for On class loading, one could even track whether there is any override in the first place. And only if that's the case, use the variant that needs to check. Most of the pieces are there. I fear it will take an external stimulus, like someone one the internet just being annoying enough to make me feel like I have to proof that it works 😉 |
|
Annoying? Moi? I don't mind what we do, so long as it's the same for all SOMs, and I know I'm comparing apples to apples. Since I'm stupid and lazy, I'd rather not implement |
|
I am not sure what exactly you imagine with apples to apples. You did your own design without really looking at the existing implementations, I believe. As far as I can tell, your design differs in important aspects. I am not sure what other differences there are, but apples to apples seems more likely between TruffleSOM and PySOM, which have relatively few differences, mostly motivated by the underlying implementation platform. |
|
There are, of course, always many types of apples, but I think that we can probably agree that having different implementations literally executing different user code is not what we want :) Put another way: if we say that all implementations must do the SOM++ |
|
Hm. I don't know. I think, for the comparison you have in mind, it would be better to simply rip out the optimizations in question and then run the comparison that you want to do, without "mandating" what a SOM should be doing. I'd think, a SOM should only be mandated to "adhere to a specification" (which isn't existing, and very badly approximated by the unit tests) similar to ECMAScript. After all, different SOMs have different reasons for existing. Some exist just for the joy of it, like SOM-RS or JsSOM... |
OK. I also realised (sorry it's been a busy day!) that I missed some bits:
I guess this is your Unix stack limit?
Eek. Dunno! If you have any suggestions, I'm all ears! |
Ehm, no. This is your SOM stack, which is about to overflow here: https://github.com/softdevteam/yksom/blob/master/src/lib/vm/core.rs#L403-L405 If I glance at the code correctly, I'd need to increase this
Yeah. This is more hairy to debug. I'd probably start with printing out computational results in the benchmark code and diff the log between ykSOM and SOM-java perhaps to see where they start to diverge to narrow down what the cause could be. |
I'd forgotten about that! I wonder what a safe limit might be? I'd be fine with doubling or quadrupling it if you think that's necessary. |
I use this in TruffleSOM... Not, well, ideal... No idea how this would translate to your case. I'd need to test. But not today. |
|
Oh, interesting, you have to increase the Java stack size! At some point you could definitely also explode the Rust stack too, but my experience is that most shells set that to a fairly high value, so... hopefully not! |
|
Just trying to run the benchmarks again, I ended up with the below. Less might work, but I didn't have the patience. diff --git a/src/lib/vm/somstack.rs b/src/lib/vm/somstack.rs
index b231415..1a3c35c 100644
--- a/src/lib/vm/somstack.rs
+++ b/src/lib/vm/somstack.rs
@@ -4,7 +4,7 @@ use std::gc::Gc;
use crate::vm::{objects::NormalArray, val::Val};
-pub const SOM_STACK_LEN: usize = 4096;
+pub const SOM_STACK_LEN: usize = 4096 * 16;
/// A contiguous stack of SOM values. This stack does minimal or no checking on important
/// operations and users must ensure that they obey the constraints on each function herein, or
|
|
Yeah, that value is a bit too small in retrospect. I'll update it to something bigger shortly. |
|
CD doesn't give correct results, probably as above, but the current results:
|
I finally setup yksom to run the Are We Fast Yet benchmarks: http://github.com/smarr/are-we-fast-yet
CD and Havlak are failing.
Havlak seems not unusual:
So, is there a way to easily increase the available stack?
CD will require some debugging.
It runs into a division by zero, which likely means that there's some arithmetic operation that's not quite right:
The text was updated successfully, but these errors were encountered: