tag:blogger.com,1999:blog-5217445496019828087.comments2019-04-25T06:07:46.857-05:00pybox2dknehttp://www.blogger.com/profile/01886171929967978645noreply@blogger.comBlogger131125tag:blogger.com,1999:blog-5217445496019828087.post-20742537609722754562012-01-07T10:33:10.521-06:002012-01-07T10:33:10.521-06:00Hi Xistic,
I fully realized from the start that t...Hi Xistic,<br /><br />I fully realized from the start that the pure python port would be an exercise in futility, at least from the standpoint of having a full-speed port. However, I really wanted to see how it would turn out -- and inspiration like that is not something to ignore. In no real way did I expect it to be a replacement for some sort of wrapped port (although I can't say I didn't dream of that possibility). I think it's also a fairly clear port that people wondering about the internals of Box2D could take a look at.<br /><br />I fully agree that moving away from SWIG is the right way to go. Whether it be Cython, SIP, Pyrex, or whatever, they would all be much better suited than SWIG. Were I to do it, I think the interface would be similar to the pure python port.<br /><br />However, the question is this: is there even a sufficient user base to make the effort of redoing the library meaningful? I don't see many new projects including pybox2d, and those that do seem to use the old versions.<br /><br />Perhaps the reason for that is that from version to version there have been complaints that my attempts to improve things break backward compatibility -- which as a library user I can understand. At the same time, though, this leaves me with a set of poor decisions I made years ago when I hadn't a clue about releasing libraries.<br /><br />If minor backward compatibility issues are enough to ignore the newer versions, I can't imagine what the response would be to a completely rewritten library.knehttps://www.blogger.com/profile/01886171929967978645noreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-61199293169884414802012-01-05T18:39:16.095-06:002012-01-05T18:39:16.095-06:00kne,
Have you considered using Cython as a replac...kne,<br /><br />Have you considered using Cython as a replacement for swig instead of re-implementing the core bits of box2d? <br /><br />I understand why you dislike swig. I don't like it because you are kind of stuck with the interface that it gives you and you have an ugly mass of interface files. The interface is often less than ideal and cannot properly handle some circumstances. <br /><br />Wrapping C++ libraries in Cython is easy. This method is less painful that it seems. Crossing the python/C++ boundary is pretty straightforward. See a good example use case here: http://www.panda3d.org/blog/?p=173<br /><br />Also I would be interested in what you did as far as adding Cython keywords to help speed up your code in your first attempt. Unless you are willing to heavily mark up your code with Cython types you are not going to see much more than a 30% improvement. If you are concerned with still running your code in "pure python" mode you can use Cython's decorators to make both possible.<br /><br />If it were me I would use Cython to write a new wrapper from scratch so I could create a more python-like interface to the C++ objects.<br /><br />I really hate to see you duplicating the efforts of box2D with a performance penalty.KyleMhttps://www.blogger.com/profile/10154623130812820415noreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-19325191962291980442011-11-19T09:35:06.958-06:002011-11-19T09:35:06.958-06:00mgo12,
I wish I could get ShedSkin working with i...mgo12,<br /><br />I wish I could get ShedSkin working with it! If you can, let me know.<br /><br />Fahri,<br /><br />Several threads on the Box2D forums have mentioned it before. That is definitely something I would like to investigate at some point.<br /><br />Also, I've been thinking that I could potentially use multiprocessing's shared lists (or maybe even proxies, somehow?) to side-step serialization of the objects. I'll have to look to see if anyone's already done any benchmarks before I spend my own time.knehttps://www.blogger.com/profile/01886171929967978645noreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-44419484236690707292011-11-18T21:23:03.591-06:002011-11-18T21:23:03.591-06:00Here is an idea about parallel computing. Have yo...Here is an idea about parallel computing. Have you considered assigning each "island" to individual processors (or processes)? Since each island is an independent group of elements (they are either connected by connectors or in contact with each other) they are perfect candidates for parallel computing. Would book keeping, inter-communication, and assembly/disassembly cost be high enough to offset the savings?Fahri Basegmezhttps://www.blogger.com/profile/09991247936241412894noreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-70895184636013666152011-09-09T21:32:47.419-05:002011-09-09T21:32:47.419-05:00You might want to try ShedSkin (http://code.google...You might want to try ShedSkin (http://code.google.com/p/shedskin/) it's pretty fast (even faster than pypy on several benchmarks)Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-81467269132325056162011-08-25T10:28:48.134-05:002011-08-25T10:28:48.134-05:00Hi Fahri,
As you say, it requires vectorization t...Hi Fahri,<br /><br />As you say, it requires vectorization to get the speed benefits of Numpy. That would require a *great* deal of refactoring, which I unfortunately did not (and do not) have the time for. Without vectorization, I highly doubt there is any significant performance difference with my little module.<br /><br />As it is, the pure Python port is a (mostly) 1:1 conversion to Python, with some Python niceties as I saw fit. The addition of the C module was primarily for learning purposes -- as it was my first pure C extension -- and secondarily just to see how much faster it could be.<br /><br />I'd be very happy to see someone else modify the engine to work with Numpy (or my preferred <a href="https://bitbucket.org/caseman/planar/" rel="nofollow">Planar</a>).<br /><br />Myself, however, I'm afraid I've reached the limit of my inspiration for now.<br /><br />Kenknehttps://www.blogger.com/profile/01886171929967978645noreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-46833046837785985482011-08-24T22:58:37.120-05:002011-08-24T22:58:37.120-05:00Hi Ken,
Wow, this looks very impressive.
I gott...Hi Ken,<br /><br />Wow, this looks very impressive. <br /><br />I gotta admit I haven't checked your C extensions but IMHO Numpy would be the best way to go in terms of speed, accuracy and robustness. Testing and verification is the most difficult part of creating scientific/math libraries and Numpy is well tested and verified.<br />Documentation is another hassle and Numpy shines here too.<br /><br />If your users are going to use the library for educational purposes then exposing Numpy out of the box would provide an excellent environment for many different use cases.<br /><br />I am not sure how you tested Numpy but instead of using loops if you vectorize the data then Numpy should be as fast as the C++ version of Box2D if not faster. Years ago I have tested Numpy against some C and Fortran libraries and results were very good. I wouldn't be surprised if it got even faster since then.<br /><br />If your primary goal is to keep the library "small" then it may be possible to compile a subset of Numpy (only matrix and vector algebra modules perhaps?).<br /><br />As you can see I am a big fan of Numpy and it seems like size is the only downside of using it. <br /><br />FahriFahri Basegmezhttps://www.blogger.com/profile/09991247936241412894noreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-51935276451518894642011-08-18T10:29:11.871-05:002011-08-18T10:29:11.871-05:00It's always nice to hear that. :)It's always nice to hear that. :)knehttps://www.blogger.com/profile/01886171929967978645noreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-18527155871841824982011-08-16T18:29:52.141-05:002011-08-16T18:29:52.141-05:00Just want to write again and say how much *fun* I&...Just want to write again and say how much *fun* I'm having with PyBox2D! Your examples give me lots of ideas to places to explore too! Thank you!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-88928611201675639432011-06-18T08:45:27.556-05:002011-06-18T08:45:27.556-05:00Thanks for checking it out, michael. :)
Porting m...Thanks for checking it out, michael. :)<br /><br />Porting more things to C seems to be a logical step, but what exactly should be ported is no longer as clear as it was with the vector (etc.) types. Everything that remains is tied together fairly tightly.<br /><br />I still have yet to get a line profiler (as in one that shows how long each line is taking instead of just function calls) working. It might help shed some light on what's happening in the "empty areas" in the runsnakerun graphs.knehttps://www.blogger.com/profile/01886171929967978645noreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-1765343815522925792011-06-17T20:44:03.543-05:002011-06-17T20:44:03.543-05:00Excellent detective work. I think creating pure C ...Excellent detective work. I think creating pure C extensions to keep it light and focused was a good choice in this situation. I looked at the C extension code and it's very clear. I like hearing "performance significantly improved"!<br /><br />That's interesting to hear your results from using multiprocessing. I wish I had some ideas of where to look next...<br /><br />Didn't know about runsnakerun--it *is* fantastic, thanks for mentioning it!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-63591207478672220282011-06-06T21:16:02.431-05:002011-06-06T21:16:02.431-05:00andy,
Thanks for the comment. I'm hoping it w...andy,<br /><br />Thanks for the comment. I'm hoping it will eventually be a viable option. Have you tried pymunk on that platform? I'd imagine it has less portability issues than pybox2d.<br /><br />In any case, I've put my first attempt at going "mostly Python" (with a bit of C) route up on the SVN, you'll find the short changelog <a href="http://code.google.com/p/pybox2d/source/detail?r=341" rel="nofollow">here</a>. The C portion may be a bit messy, but I'm new at pure C extensions, so go easy on me. :)<br /><br />It's significantly faster in parts, but still not quite enough for things like the pyramid test and such. I also have yet to test on Linux, so there are likely to be issues with compilation. Please let me know if you run into any.<br /><br />% svn checkout http://pybox2d.googlecode.com/svn/branches/pure_python/ pypybox2dknehttps://www.blogger.com/profile/01886171929967978645noreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-74157508517489453832011-06-05T00:20:31.045-05:002011-06-05T00:20:31.045-05:00Would definitely be interested in the mostly-pytho...Would definitely be interested in the mostly-python version of pypybox2d. Looks like it will suit applications where pybox2d is difficult to get working (e.g. kivy on android), even if performance takes a respectable hit.<br /><br />Looking forward to seeing where this part of the project goes.andyhttps://www.blogger.com/profile/17391872069658181784noreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-53611826933339825172011-06-03T23:02:55.212-05:002011-06-03T23:02:55.212-05:00Just a quick update... on not-so-pure Python physi...Just a quick update... on not-so-pure Python physics:<br /><br />So I'm making my first attempts at pure C Python types. My inspiration for trying this out, Vec2 and Mat22, were surprisingly easy to write. Without any modification to the rest of the library, changing these two classes have made for a very significant increase in speed (and if they didn't, I'd be rather confused).<br /><br />It's interesting to watch the <a href="http://www.vrplumber.com/programming/runsnakerun/" rel="nofollow">runsnakerun</a> squares shrink away after porting some core functions back to C.<br /><br />Anyway, it's rough around the edges, and still somewhat of a separate project from 'pypybox2d'. I'll put it up perhaps on a separate branch on the SVN sometime soon if anyone's interested.<br /><br />Perhaps there's some future in a mostly-Python physics library? Probably not, but it's fun to play around with, at least!knehttps://www.blogger.com/profile/01886171929967978645noreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-59745000997401081852011-05-28T07:14:42.142-05:002011-05-28T07:14:42.142-05:00Thanks for the comments, michael and dabski. I can...Thanks for the comments, michael and dabski. I can't say I'm a big fan of "soft pybox2d", somehow...<br /><br />thouis:<br />That's too bad about it not being supported directly. Using normal threads, I can imagine the code would be that much uglier. Thanks for looking into doing common.py. I think that with some type annotations there will be significant speedups over pure Python. I look forward to seeing what you come up with!knehttps://www.blogger.com/profile/01886171929967978645noreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-7450871594969558022011-05-26T03:25:58.061-05:002011-05-26T03:25:58.061-05:00Cython doesn't support parallelism directly, b...Cython doesn't support parallelism directly, but it is fairly easy to mark code as not requiring the GIL, allowing multiple python threads to be in the same code block at the same time.<br /><br />I'll take a look at cythonizing common.py, and let you know how it goes.thouishttps://www.blogger.com/profile/08805809527656089305noreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-3875662606668913972011-05-24T19:20:05.714-05:002011-05-24T19:20:05.714-05:00Incredible work, and the PyPy speedups are stunnin...Incredible work, and the PyPy speedups are stunning! This is very exciting! <br /><br />As for a name... how about "Soft PyBox2D"?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-46083677752680352012011-05-24T07:33:33.417-05:002011-05-24T07:33:33.417-05:00This sounds awesome kne. I look forward to a SWIGl...This sounds awesome kne. I look forward to a SWIGless pybox2d with no more py2exe issues.dabskihttps://www.blogger.com/profile/07901452043015337082noreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-18990802785802364372011-05-23T08:46:43.688-05:002011-05-23T08:46:43.688-05:00thouis, that would be great. I unfortunately haven...thouis, that would be great. I unfortunately haven't had too much time to learn Cython yet.<br /><br />My one simple test was just converting common.py (it does all of the vector calculations and such) into a compiled Cython module. It gave strange results once, and worked fine the next. Bringing all of the modules together into a Cython library would be an interesting next step.<br /><br />Also, if there's any support for simple parallel processing in Cython (since, as far as I understand, we could side-step the GIL), it would be very interesting to parallelize some of the calculations.knehttps://www.blogger.com/profile/01886171929967978645noreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-21196680540202123632011-05-22T13:41:37.862-05:002011-05-22T13:41:37.862-05:00I'd be willing to help with Cython. I've ...I'd be willing to help with Cython. I've done a reasonable amount of coding in it, including a few projects to wrap existing libraries.thouishttps://www.blogger.com/profile/08805809527656089305noreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-41527445870597703252011-05-20T17:06:43.853-05:002011-05-20T17:06:43.853-05:00Sure, I understand. I'll definitely be testing...Sure, I understand. I'll definitely be testing it with every new release of PyPy to see how it's progressing.knehttps://www.blogger.com/profile/01886171929967978645noreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-78744810655386382682011-05-20T14:52:50.552-05:002011-05-20T14:52:50.552-05:00note that, although on the long run pypy is faster...note that, although on the long run pypy is faster, we are still not satisfied with the result. We should do much better, and have a faster warmup time. PyPy is full of heuristics to determine what to compile and when, and apparently they just don't work too well with this example. Hopefully, next versions of PyPy will be better :-)Antonio Cunihttps://www.blogger.com/profile/17017456817083804792noreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-25868535690537253712011-05-20T11:06:29.393-05:002011-05-20T11:06:29.393-05:00Very interesting, Antonio. I'll admit my orig...Very interesting, Antonio. I'll admit my original PyPy tests were rather artificial. It does make sense that the JIT needs some time to "warm up".<br /><br />I'll change it around to remove the numbers.Number dependency. Breaking up the functions further would even allow the removal of isinstance in some cases, at the expense of some ease of use for the vector/matrix classes.<br /><br />Thanks for the reply. :)knehttps://www.blogger.com/profile/01886171929967978645noreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-31607481996256119082011-05-20T09:21:16.883-05:002011-05-20T09:21:16.883-05:00I tried to benchmark this against PyPy, using a mo...I tried to benchmark this against PyPy, using a modified version of hello.py<br />which does an arbitrary number of steps().<br /><br />I benchmarked the pure python implementation both on CPython 2.7.1 and PyPy<br />fa691b4a848a (which is more or less the same as 1.5, with few improvements).<br /><br />Here are the results (in seconds) for an increasing number of iterations:<br /><br />CPython 2.7.1:<br />100: 0.01<br />1000: 0.14<br />10000: 1.3<br />100000: 13.19<br />1000000: 131.54<br /><br />PyPy<br />100: 0.05<br />1000: 0.5<br />10000: 2.06<br />100000: 6.54<br />1000000: 51.22<br /><br />As you can see, the JIT takes longer to warmup, but at the end it produces<br />better code.<br /><br />Then, I looked at the code produced by the PyPy JIT, and saw that it spent a<br />lot of time inside abc.py, which is caused by the "isinstance(other,<br />numbers.Number)" calls in common.py. It turns out that the caching done by<br />__instancecheck__ is very JIT-unfriendly. So, I removed the reference to ABCs<br />by adding this line to the top of common.py:<br /><br />numbers.Number = (float, int, long)<br /><br />On CPython, it gives a small improvement (~12%), but on PyPy it makes it 2.3<br />times faster!<br /><br />CPython 2.7.1 w/o ABCs<br />100: 0.01<br />1000: 0.12<br />10000: 1.16<br />100000: 11.81<br />1000000: 117.39<br /><br />PyPy w/o ABCs<br />100: 0.04<br />1000: 0.44<br />10000: 1.7<br />100000: 3.64<br />1000000: 22.25<br /><br />Finally, I started to play a bit with the various JIT parameters, and found a<br />way to improve performances even further, by passing "--jit trace_limit=20000" to PyPy:<br /><br />PyPy w/o ABCs --jit trace_limit=20000<br />100: 0.05 <br />1000: 0.46 <br />10000: 1.61 <br />100000: 3.03 <br />1000000: 16.83 <br /><br />So, at then end of the day, on the long run PyPy is at least 7.8 times faster<br />than CPython. Still a lot slower than the C++ version, though :-(Antonio Cunihttps://www.blogger.com/profile/17017456817083804792noreply@blogger.comtag:blogger.com,1999:blog-5217445496019828087.post-15574857332187174452011-05-05T19:33:15.777-05:002011-05-05T19:33:15.777-05:00I appreciate it, Bob. Real life does get in the wa...I appreciate it, Bob. Real life does get in the way a lot, doesn't it? :)knehttps://www.blogger.com/profile/01886171929967978645noreply@blogger.com