Replies: 3 comments 3 replies
-
|
This is how it currently handles Java. Iirc there is still code in the native decompiler code base to explicitly handle it so I think it would be required for other languages as well. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for sharing what you know. I looked at the code in links suggested and also looked at Come Get Your Free NSA Reverse Engineering Tool. What I can figure out from this is that the impedance mismatch between Python bytecode and what is currently in Ghidra is so great that it would be foolhardy for anyone to try to undertake this except as a full-time dedicated activity. And even there, I wouldn't expect much. It also doesn't really address the questions I have... Is there interest in decompiling Python bytecode?I'll take the lack of response not I've gotten here and elsewhere as a: "no, not really". In my own first-hand experience was that back in 2020 three people from Microsoft contacted me about some malware written in Python 2.7 that they needed to decompile. The good news there is that for Python 2.7, decompilers are readily available. For current Python: 3.9 and 3.10, not yet. But the microsoft security team was more of a one-of-the-kind occurrence than something periodic. A not uncommon attitude is that if it worked out magically somehow the last time, it will work out the next time as well. And if it doesn't, then is the time to start thinking about adjusting interest. Other than that thing from 2020 and hearing vague reports that some malware written is written in Python (which might be on the uptick), I guess things have been quiet so far as I know. Is there interest more broadly in decompiling Dynamic High-level languages in general?Again, I would assume no since if there were, that interest would appear in Python. Is there any research going on in decompilers?I am not finding anything showing up that isn't continuation of stuff a decade old or more. And within that time, I have come to realize that the area of decompilation in dynamic high-level languages hasn't been studied to any extent as a class in of itself. I have also come to realize that decompilers for this class are a bit different from the general-purpose kind of decompiler that appears in Ghidra. There are many decompilers out there for such specific high-level languages, but the ones I've looked at use a seat-of-the-pants approach My own experience leads me to believe much that could be done, if there was interest in reducing the ad-hoc-ness of this kind of decompiler. But again, right now this stuff is a bit labor intensive which means you don't expect a lot of activity to appear without some motivation. I've written about the difference between general-purpose decompilers and high-level dynamic language decompilers here. Lastly with respect to research on decompilers: there were some university assistant professors and their grad students that contacted a few months back about a paper they hoped to present based on the decompilers I've worked on. But like the microsoft security team incident this is more of a one-of-a-kind activity than something that happens time to time. |
Beta Was this translation helpful? Give feedback.
-
Sorry I was just trying to point you in, what I thought would be, the right direction. Your best bet for a solid answer would most likely have to be from someone on the Ghidra team. Considering that the decompiler is just a c++ program communicating with the java end by passing xml around I feel like it might be more reasonable to make a Ghidra plugin which does something similar and uses your existing python decompiler. |
Beta Was this translation helpful? Give feedback.
-
No need to be sorry. As I said, I appreciate the response and that you are trying to be helpful. Sometimes this is a rare commodity. Elsewhere I wrote about an experience I remember and cherish precisely because it is an example of a situation where having someone willing to help was a success even though he had not prior detailed knowledge or experience. However, for every story like this there are innumerable others that go the other way. Actual facts do matter, as is likely the case here. Here, I'd like to add one other observation. I get that in the secrecy business, you never want to say (or admit) what you don't know or where you are weak. I worked at IBM for 12 years, and some of the dumbest ideas I ever came across were classified at the highest levels of confidentiality. At IBM Research the inside joke was that if you had a really dumb idea, just put the highest level of security classification and you are covered — some people will think you really smart. Similarly, I've looked at the other Python decompilers out there to try to understand more about them. Other than actual code, you won't find much written about how they work (even as comments inside the program) let alone what their strengths and weaknesses are. Although I realize first hand that it takes a lot effort to try to explain something like this, I sometimes wonder if there isn't also a little of this effect going on - don't say anything and then people will given you the benefit of the doubt of knowing something you might not know, or understand only intuitively. On the other side, there is a school of thought in Science, Mathematics and Engineering that is open about sharing information because that is the common and accepted way that things become more understood and improved: by sharing, others assist or build upon that, expose, and remove weaknesses.
Sure, anything is possible if we just think about things at a high-enough level and in the abstract. We all understand English here, and I can point you to the Python reference manual which describes the programming language; in there you'll find a detailed description of an internal representation or AST of that along with a somewhat detailed description of individual Python bytecode instructions and what they do. This is all in English with formal grammars. And if this isn't enough, there is the actual Python code which is largely written in C and I have a toy bytecode interpreter written in Python if you don't want to understand the C code or would like a more interactive way to step instructions. So problem solved, right? SMOP (small matter of programming) Not really. As I said, I think there is an impedance problem at a somewhat fundamental level. Here is an analgous sitiation which is also relevant because Ghidra provides a "common debugger"... Have you never noticed that debuggers for such high-level dynamic languages such as those for Perl, Python, Ruby, Lisp, Javascript have these things that work differently from low-level debuggers like LLDB and GDB? Out of the box, LLDB and GDB don't directly support any of debugging any of these languages. In some cases though there are what feels to me more like fragile hacks that allow the lower level debugger to hook back into the high-level language interpreter which does the heavy lifting. But as I said, these are fragile, very custom, and seem more the exception than the rule. And it is just not the way programmers typically work when they want to debug a program in such a language, except when it is absolutely necessary, and not by choice. Some other some specific differences you'll notice between low-level debuggers and high-level debuggers: a somewhat superficial thing right when you enter the debugger by calling it from the outset, is whether the program is already in progress or not. For low-level debuggers that is not the case. For high-level debuggers it generally is the case that when you've entered the debugger the program is usually already in progress. And if you are writing a "common" front-end interface for debuggers such as I have worked on in ddd and realgud, this is an important distinction to understand. In low level debuggers, seeing the high-level state of a programming-language object and being able to evaluate arbitrary expressions in the language (outside of possibly making functions calls) is kind of hard to do while in a high-level debugger it is easy to do since the debugger is written in the language it is debugging and their for is on par with the programming language's expression evaluation. But on the other side, stepping a machine instruction, stepping a bytecode instruction, setting a hardware breakpoint even when there is hardware support for it, is hard to do if it can be done at all. Is this accidental or fundamental? I would say fundamental. And this is the kind of thing I mean by "impedance mismatch". The details for decompilation is different, but it's the same kind of idea. Lastly on the topic of opining about stuff at a broad level where everything is simple if you don't need to consider the details, if find it interesting to look at the discussion here to the question asked "Have you ever made a decompiler in Python, and how did you make it?". There are 7 answers. 5 of the people responding say up front that they haven't written such a thing (or a any sort of decompiler) but gave what I imagine they thought were reasonable answers or conjectures. And if you were not to consider the details, I agree the answers seem pretty decent and reasonable. However having more understanding of the topic, you come to realize that what seems like a reasonable idea has certain problems or flaws. Aside from my suggested answer, the other person who had at least tried to write a decompiler had, in my opinion, the best insight and advice of the bunch. And if I had no prior knowledge with all of the noise of the other 5 answers, I might be hard pressed to assess whether his answer was the best among those who hadn't tried even to write such a decompiler. In short, sadly, details do matter. But to close on a positive note, thanks for your reply. I appreciate it and thanks for taking the time. |
Beta Was this translation helpful? Give feedback.
-
|
Not sure if it is of interest, but I came across this when looking for a way to decompile Javascript (V8 bytecode) so figured I'd link that approach: https://swarm.ptsecurity.com/how-we-bypassed-bytenode-and-decompiled-node-js-bytecode-in-ghidra/ |
Beta Was this translation helpful? Give feedback.
-
Thanks! It is of interest because I am interested in the different kinds of custom bytecode that exist and the decompilation techniques behind them. And this definitely includes V8 bytecode.
That is useful to know. As the title suggests, I was also asking about interest in technology for high-level dynamic decompiling. The lack of response has answered that question pretty clearly, to my mind. Or at least with respect to the community that follows discussions here. And even if there were interest, the next obvious follow-on question would have been about interest more than the kind of thing where you +1 or -1 something but are willing to commit to do something about it. Looking at the write up linked and looking at ghidra code, it seems clear to me that implementing this as a ghidra plugin is not easy and would take a bit of work. Far more work than someone like myself who has done this on the side as a hobby would be interested in. |
Beta Was this translation helpful? Give feedback.
-
How does ghidra handle decompiling high-level languages that compile to their own internal representation?
There is a class of languages that work like Python where:
Possibly WebGL, Java, Lua and others fall into this class. GNU Emacs and Ruby I think most definitely do.
I've been interested in the Python case for a while and the method I've been using seems to be more applicable outside of Python and even what I will call "general-purpose" decompilers. Although I think there are ideas that are very amenable to machine learning, all of this is a bit labor intensive. And that's why I haven't strayed far outside of Python.
But I wondered if there was any research done here. I know from the reverse engineering side there has been a lot of interest in Python decompilation.
I also understand that there has been (still is?) malware written in Python. Is that it or are there other programming languages used for which decompilers are in need?
Beta Was this translation helpful? Give feedback.
All reactions