Quantcast

QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

classic Classic list List threaded Threaded
70 messages Options
1234
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

John Spackman-3
Hi all

There is a first release of my QxCompiler that adds ES6 to Qooxdoo applications and replaces the generate.py toolchain with a faster, 100% Javascript tool that is easily extensible.

You can find the first release at GitHub here: https://github.com/johnspackman/qxcompiler

It’s an alpha release, but something that’s been in development for a while now and which I’m starting to build into my production servers.  

I’m very open to pull requests or collaboration, and keen to see this become a useful tool for myself as well as others so any questions etc please ask.

Regards
John





------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
qooxdoo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

mitya
John, congratulations with the long awaited release! :)

qooxdoo guys, do you think that project like this could at some moment
land in qooxdoo and get official support? What about qooxdoo patches
(mostly strict mode compatibility related, AFAIK) - could they be
merged upstream? This would reduce maintainship costs for those who
want to experiment with both QxCompiler and official toolchain.

I didn't yet try QxCompiler with my project - I feel I yet lack some
understanding of how it works. However, I tried to test it with the
skeleton app. Off the top of my head:

- one needs to do "npm install" first and to run test scripts with
"node <script>.js". This might be quite obvious for those experienced
with Node, but I guess the docs will nevertheless benefit from
mentioning this;
- it's not easy to clone a repo unless you've set up Github SSH access.
This is because of "qooxdoo" submodule pointing to "[hidden email]:john
spackman/qooxdoo.git". Could it be a HTTPS URL instead?
- does QxCompiler honour QOOXDOO_PATH setting? How do I define qooxdoo
location if it is different from the bundled one?
- testdata/qxt directory doesn't contain skeleton app. In order to play
with test scripts, one needs to create the app manually (as
"skeleton.Application"?) and copy it to the said location;
- does lib/compiler.jar really belong there?

Most importantly, I was unable to find clear instructions how to
produce a single-script minified build (a-la "generate.py build"). By
deafult, QxCompiler produces a bunch of JS files, which significantly
slows down loading and, obviously, is not acceptable for production. Is
it possible at all with the current version (maybe in combination with
classic generator)?

I hope these are just minor issues. After all, you've done a great job
:) I wish you good luck and further progress wih QxCompiler.

Dimitri

> Hi all
>
> There is a first release of my QxCompiler that adds ES6 to Qooxdoo
> applications and replaces the generate.py toolchain with a faster,
> 100% Javascript tool that is easily extensible.
>
> You can find the first release at GitHub here: https://github.com/joh
> nspackman/qxcompiler
>
> It’s an alpha release, but something that’s been in development for a
> while now and which I’m starting to build into my production
> servers.  
>
> I’m very open to pull requests or collaboration, and keen to see this
> become a useful tool for myself as well as others so any questions
> etc please ask.
>
> Regards
> John
>
>
>
>
>
> -------------------------------------------------------------------
> -----------
> Site24x7 APM Insight: Get Deep Visibility into Application
> Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
> _______________________________________________
> qooxdoo-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
qooxdoo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

John Spackman
Hi Dimitri
 
Thanks for checking out so quickly :)
 
Re: npm and docs: done
 
Re: qooxdoo submodule and https: fixed
 
Re: QOOXDOO_PATH: no - that's a generator.py/config.json setting and config.json files are completely ignored; when using the QxCompiler API, once you have told it where the Qooxdoo library is you have done the equivalent of setting QOOXDOO_PATH.  Of course, as it's API based you can define variables etc as suits you best :)
 
In test/compile-app-demo.js it adds the Qooxdoo library like this:
    maker.addLibrary("../qooxdoo/framework", cb);
 
Just change the path to point to your preferred location for qooxdoo.  To make this a bit clearer, I've just modified the test/compile-app-demo.js to declare a variable QOOXDOO_PATH
 
I expect that the command line version will imply have a search path for libraries and just auto-discover by searching for Manifest.json files.
 
Re: missing testdata/qxt: something went wrong with your checkout, that directory is definitely there.
 
Re: build (and source-hybrid) targets: to control output for Source vs Build vs Source-Hybrid, the API classes qxcompiler.targets.SourceTarget, qxcompiler.targets.BuildTarget, and qxcompiler.targets.SourceHybridTarget are used; test/compile-app-demo.js uses the SourceTarget but you should be able to switch it easily enough.  I say "should" because I realise now that I havn't tested them for a few weeks now and I should give them the once over ASAP :)  I've got to go out shortly so I'll take a look this evening.
 
John
 
 

From: "Dimitri" <[hidden email]>
Sent: Sunday, February 14, 2016 2:27 AM
To: "qooxdoo Development" <[hidden email]>
Subject: Re: [qooxdoo-devel] QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications
 
John, congratulations with the long awaited release! :) qooxdoo guys, do you think that project like this could at some moment land in qooxdoo and get official support? What about qooxdoo patches (mostly strict mode compatibility related, AFAIK) - could they be merged upstream? This would reduce maintainship costs for those who want to experiment with both QxCompiler and official toolchain. I didn't yet try QxCompiler with my project - I feel I yet lack some understanding of how it works. However, I tried to test it with the skeleton app. Off the top of my head: - one needs to do "npm install" first and to run test scripts with "node <script>.js". This might be quite obvious for those experienced with Node, but I guess the docs will nevertheless benefit from mentioning this; - it's not easy to clone a repo unless you've set up Github SSH access. This is because of "qooxdoo" submodule pointing to "[hidden email]:john spackman/qooxdoo.git". Could it be a HTTPS URL instead? - does QxCompiler honour QOOXDOO_PATH setting? How do I define qooxdoo location if it is different from the bundled one? - testdata/qxt directory doesn't contain skeleton app. In order to play with test scripts, one needs to create the app manually (as "skeleton.Application"?) and copy it to the said location; - does lib/compiler.jar really belong there? Most importantly, I was unable to find clear instructions how to produce a single-script minified build (a-la "generate.py build"). By deafult, QxCompiler produces a bunch of JS files, which significantly slows down loading and, obviously, is not acceptable for production. Is it possible at all with the current version (maybe in combination with classic generator)? I hope these are just minor issues. After all, you've done a great job :) I wish you good luck and further progress wih QxCompiler. Dimitri > Hi all > > There is a first release of my QxCompiler that adds ES6 to Qooxdoo > applications and replaces the generate.py toolchain with a faster, > 100% Javascript tool that is easily extensible. > > You can find the first release at GitHub here: https://github.com/joh > nspackman/qxcompiler > > It’s an alpha release, but something that’s been in development for a > while now and which I’m starting to build into my production > servers.   > > I’m very open to pull requests or collaboration, and keen to see this > become a useful tool for myself as well as others so any questions > etc please ask. > > Regards > John > > > > > > ------------------------------------------------------------------- > ----------- > Site24x7 APM Insight: Get Deep Visibility into Application > Performance > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month > Monitor end-to-end web transactions and take corrective actions now > Troubleshoot faster and improve end-user experience. Signup Now! > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > _______________________________________________ > qooxdoo-devel mailing list > [hidden email] > https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 _______________________________________________ qooxdoo-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
qooxdoo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

John Spackman
Further to this - I've added some docs about the API, but also seen that the BuildTarget has a bug and doesn't work - it's only a minor issue but I probably wont have time to fix it until tonight
 
John
 
 
 

From: "John Spackman" <[hidden email]>
Sent: Sunday, February 14, 2016 9:50 AM
To: "qooxdoo Development" <[hidden email]>
Subject: Re: [qooxdoo-devel] QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications
 
Hi Dimitri
 
Thanks for checking out so quickly :)
 
Re: npm and docs: done
 
Re: qooxdoo submodule and https: fixed
 
Re: QOOXDOO_PATH: no - that's a generator.py/config.json setting and config.json files are completely ignored; when using the QxCompiler API, once you have told it where the Qooxdoo library is you have done the equivalent of setting QOOXDOO_PATH.  Of course, as it's API based you can define variables etc as suits you best :)
 
In test/compile-app-demo.js it adds the Qooxdoo library like this:
    maker.addLibrary("../qooxdoo/framework", cb);
 
Just change the path to point to your preferred location for qooxdoo.  To make this a bit clearer, I've just modified the test/compile-app-demo.js to declare a variable QOOXDOO_PATH
 
I expect that the command line version will imply have a search path for libraries and just auto-discover by searching for Manifest.json files.
 
Re: missing testdata/qxt: something went wrong with your checkout, that directory is definitely there.
 
Re: build (and source-hybrid) targets: to control output for Source vs Build vs Source-Hybrid, the API classes qxcompiler.targets.SourceTarget, qxcompiler.targets.BuildTarget, and qxcompiler.targets.SourceHybridTarget are used; test/compile-app-demo.js uses the SourceTarget but you should be able to switch it easily enough.  I say "should" because I realise now that I havn't tested them for a few weeks now and I should give them the once over ASAP :)  I've got to go out shortly so I'll take a look this evening.
 
John
 
 

From: "Dimitri" <[hidden email]>
Sent: Sunday, February 14, 2016 2:27 AM
To: "qooxdoo Development" <[hidden email]>
Subject: Re: [qooxdoo-devel] QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications
 
John, congratulations with the long awaited release! :) qooxdoo guys, do you think that project like this could at some moment land in qooxdoo and get official support? What about qooxdoo patches (mostly strict mode compatibility related, AFAIK) - could they be merged upstream? This would reduce maintainship costs for those who want to experiment with both QxCompiler and official toolchain. I didn't yet try QxCompiler with my project - I feel I yet lack some understanding of how it works. However, I tried to test it with the skeleton app. Off the top of my head: - one needs to do "npm install" first and to run test scripts with "node <script>.js". This might be quite obvious for those experienced with Node, but I guess the docs will nevertheless benefit from mentioning this; - it's not easy to clone a repo unless you've set up Github SSH access. This is because of "qooxdoo" submodule pointing to "[hidden email]:john spackman/qooxdoo.git". Could it be a HTTPS URL instead? - does QxCompiler honour QOOXDOO_PATH setting? How do I define qooxdoo location if it is different from the bundled one? - testdata/qxt directory doesn't contain skeleton app. In order to play with test scripts, one needs to create the app manually (as "skeleton.Application"?) and copy it to the said location; - does lib/compiler.jar really belong there? Most importantly, I was unable to find clear instructions how to produce a single-script minified build (a-la "generate.py build"). By deafult, QxCompiler produces a bunch of JS files, which significantly slows down loading and, obviously, is not acceptable for production. Is it possible at all with the current version (maybe in combination with classic generator)? I hope these are just minor issues. After all, you've done a great job :) I wish you good luck and further progress wih QxCompiler. Dimitri > Hi all > > There is a first release of my QxCompiler that adds ES6 to Qooxdoo > applications and replaces the generate.py toolchain with a faster, > 100% Javascript tool that is easily extensible. > > You can find the first release at GitHub here: https://github.com/joh > nspackman/qxcompiler > > It’s an alpha release, but something that’s been in development for a > while now and which I’m starting to build into my production > servers.   > > I’m very open to pull requests or collaboration, and keen to see this > become a useful tool for myself as well as others so any questions > etc please ask. > > Regards > John > > > > > > ------------------------------------------------------------------- > ----------- > Site24x7 APM Insight: Get Deep Visibility into Application > Performance > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month > Monitor end-to-end web transactions and take corrective actions now > Troubleshoot faster and improve end-user experience. Signup Now! > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > _______________________________________________ > qooxdoo-devel mailing list > [hidden email] > https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 _______________________________________________ qooxdoo-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
qooxdoo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

mitya
Hi John! Ready for another dozen of questions and bug reports? ;)

1. As for testdata/qxt directory - yup, it is indeed present, but I don't see "source" subdirectory, where I would expect the actual application code to reside. So I just ran "create-application.py -n qxt" in another location and copied the generated "source" subdirectory to testdata/qxt. Is it how it is supposed to work?

2. The build process spits a lot of trace messages like this:
Function enter: anonymous
Function exit: anonymous

SourceTarget completes successfully, but BuildTarget fails with the following:

2016-02-15 04:10:22.045 [info ] makers          Writing target Build Target: ../testdata/qxt/build-output/
events.js:141
      throw er; // Unhandled 'error' event
      ^

Error: EMFILE: too many open files, open '../testdata/qxt/build-output//resource/qx/icon/Tango/48/actions/go-home.png'
    at Error (native)

I had to increase open file descriptors limit (ulimit -n 8192 worked for me). If this cannot be fixed/optimized, probably it should be reflected in the docs.

3. There seems to be a big mess with resources. Test app works well in its source variant, but the single-script build fails to load an icon:

000472 qx.ui.basic.Image[45-0]: Image could not be loaded: qxt/test.png boot.js:54698:13
000474 qx.ui.basic.Image[45-0]: Image could not be loaded: qxt/test.png boot.js:54698:13
000569 ImageLoader: Not recognized format of external image 'qxt/test.png'! boot.js:54698:13

Shouldn't it look in "resources" subdirectory instead? The subdir is present and does contain resources; however, it seems to contain *all* the available qooxdoo+app resources (~3600 files, ~25M). generate.py usually produces a ~150K subdir containing resources that are actually required.

At the same time, my application was unable to load resources neither in source nor in build variant. In both cases it tried to (unsuccessfully) look up resources in that very same "source-output" or "build-output" directory.

4. Our project uses your (excellent :) UploadMgr contrib. I've added the following to the async.series block:

        function(cb) {
          maker.addLibrary("/path/to/qooxdoo-contrib/UploadMgr", cb);
        },

Is this the right way to integrate a contrib into the project?

Can't yet say whether it worked or not, as I didn't manage to make application work (see below). I've noticed the following in console:
SyntaxError: in strict mode code, functions may be declared only at top level or immediately within another function (source-output/transpiled//com/zenesis/qx/upload/XhrHandler.js:212:19)
(original file line 212, function sendAsMime definition)

5. One of my classes invokes its static method from a constructor. According to qooxdoo docs, this should be done this way:
this.self(arguments).foo();

The application failed to load because of "this.constructor.self" being undefined. The following however worked fine:
this.constructor.foo();

This could potentially break applications that access static members from constructors and/or instance methods. (I'm not sure if constructor is some special case.)

6. Is there a way to pass qooxdoo environment keys to the app a la config.js? Much better if the values could be passed via command line. Our build system produces metadata (version, build number, revision, timestamp) which we are planning to pass to the app in the form of qx.core.Environment keys.

7. What about locales & translation? For us, it is essential because our app is already bilingual, and more languages are to come.

8. For our app, "generate.py build" produces a ~900K script, and it takes less than a second for the app to be up and running.
QxCompiler/BuildTarget produces a huge ~5M single script, and application startup takes up to 10 seconds (!)
What is the recommended way to produce optimized/minified build that would be comparable to the original one?

Okay, that's enough for today :) Tomorrow I'll try to figure out the cause of another failure that prevents our app from starting up. It's something related to JSON unmarshalling; the data is unmarshalled as auto-generated class (qx.data.model.prop1"prop2"prop3[123-0]), while it should be an application class resolved by a marshal delegate. This works fine with generate.py.

Good luck!
Dimitri

Further to this - I've added some docs about the API, but also seen that the BuildTarget has a bug and doesn't work - it's only a minor issue but I probably wont have time to fix it until tonight
 
John
 
 
 

From: "John Spackman" <[hidden email]>
Sent: Sunday, February 14, 2016 9:50 AM
To: "qooxdoo Development" <[hidden email]>
Subject: Re: [qooxdoo-devel] QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications
 
Hi Dimitri
 
Thanks for checking out so quickly :)
 
Re: npm and docs: done
 
Re: qooxdoo submodule and https: fixed
 
Re: QOOXDOO_PATH: no - that's a generator.py/config.json setting and config.json files are completely ignored; when using the QxCompiler API, once you have told it where the Qooxdoo library is you have done the equivalent of setting QOOXDOO_PATH.  Of course, as it's API based you can define variables etc as suits you best :)
 
In test/compile-app-demo.js it adds the Qooxdoo library like this:
    maker.addLibrary("../qooxdoo/framework", cb);
 
Just change the path to point to your preferred location for qooxdoo.  To make this a bit clearer, I've just modified the test/compile-app-demo.js to declare a variable QOOXDOO_PATH
 
I expect that the command line version will imply have a search path for libraries and just auto-discover by searching for Manifest.json files.
 
Re: missing testdata/qxt: something went wrong with your checkout, that directory is definitely there.
 
Re: build (and source-hybrid) targets: to control output for Source vs Build vs Source-Hybrid, the API classes qxcompiler.targets.SourceTarget, qxcompiler.targets.BuildTarget, and qxcompiler.targets.SourceHybridTarget are used; test/compile-app-demo.js uses the SourceTarget but you should be able to switch it easily enough.  I say "should" because I realise now that I havn't tested them for a few weeks now and I should give them the once over ASAP :)  I've got to go out shortly so I'll take a look this evening.
 
John
 
 

From: "Dimitri" <[hidden email]>
Sent: Sunday, February 14, 2016 2:27 AM
To: "qooxdoo Development" <[hidden email]>
Subject: Re: [qooxdoo-devel] QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications
 
John, congratulations with the long awaited release! :) qooxdoo guys, do you think that project like this could at some moment land in qooxdoo and get official support? What about qooxdoo patches (mostly strict mode compatibility related, AFAIK) - could they be merged upstream? This would reduce maintainship costs for those who want to experiment with both QxCompiler and official toolchain. I didn't yet try QxCompiler with my project - I feel I yet lack some understanding of how it works. However, I tried to test it with the skeleton app. Off the top of my head: - one needs to do "npm install" first and to run test scripts with "node <script>.js". This might be quite obvious for those experienced with Node, but I guess the docs will nevertheless benefit from mentioning this; - it's not easy to clone a repo unless you've set up Github SSH access. This is because of "qooxdoo" submodule pointing to "[hidden email]:john spackman/qooxdoo.git". Could it be a HTTPS URL instead? - does QxCompiler honour QOOXDOO_PATH setting? How do I define qooxdoo location if it is different from the bundled one? - testdata/qxt directory doesn't contain skeleton app. In order to play with test scripts, one needs to create the app manually (as "skeleton.Application"?) and copy it to the said location; - does lib/compiler.jar really belong there? Most importantly, I was unable to find clear instructions how to produce a single-script minified build (a-la "generate.py build"). By deafult, QxCompiler produces a bunch of JS files, which significantly slows down loading and, obviously, is not acceptable for production. Is it possible at all with the current version (maybe in combination with classic generator)? I hope these are just minor issues. After all, you've done a great job :) I wish you good luck and further progress wih QxCompiler. Dimitri > Hi all > > There is a first release of my QxCompiler that adds ES6 to Qooxdoo > applications and replaces the generate.py toolchain with a faster, > 100% Javascript tool that is easily extensible. > > You can find the first release at GitHub here: https://github.com/joh > nspackman/qxcompiler > > It’s an alpha release, but something that’s been in development for a > while now and which I’m starting to build into my production > servers.   > > I’m very open to pull requests or collaboration, and keen to see this > become a useful tool for myself as well as others so any questions > etc please ask. > > Regards > John > > > > > > ------------------------------------------------------------------- > ----------- > Site24x7 APM Insight: Get Deep Visibility into Application > Performance > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month > Monitor end-to-end web transactions and take corrective actions now > Troubleshoot faster and improve end-user experience. Signup Now! > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > _______________________________________________ > qooxdoo-devel mailing list > [hidden email] > https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 _______________________________________________ qooxdoo-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
qooxdoo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
qooxdoo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

Dietrich Streifert
In reply to this post by John Spackman-3
Hi John

Congratulations!

One question regarding this passage in the README.md:

NOTE While QxCompiler is backward compatible with generate.py and does not require changes to your code, ES6 support requires that strict mode is enabled and this can introduce a few minor compatibility issues; also, QxCompiler uses a new trick for managing dependencies and for both of these reasons you must use my fork of Qooxdoo (https://github.com/johnspackman/qooxdoo and use the qxcompiler branch). My fork is based on Qooxdoo 5.0.1.

Could you explain which changes where necessary in qooxdoo to make it work with your compiler? Would it maybe be possible to "document" those changes via a pull request against qooxdoo/qooxdoo?

Another question is related to browser compatibilities: are they changed/raised because of the ES6 and strict mode changes? Will it still run on IE8?

Regards
Dietrich



------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
qooxdoo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

John Spackman-3
Hi Dietrich

Thanks :)

There are two sets of changes – strict mode support (which is in the strict-mode-support branch), and mods needed for QxCompiler.  I will put them up as pull requests once things settle down with QxCompiler, but in the mean time both branches are based on vanilla Qoxodoo 5.0.1 (ie not including any of my other branches) so there shouldn’t be a barrier for anyone to use my branch.

Strict mode fixes are:
(a) You cannot write to properties which do not have a getter; in various places Qooxdoo will copy event properties back to the event object (which AFAICT is a fix for IE8) but this raises an exception in strict mode so needs a try/catch wrapper
(b) The global object is undefined, not “window” so if you want a global object (like “q”) it needs the “window” qualifier

As for IE8 support – I’ve not tested it but AIUI strict mode will not cause issues in IE8/IE9, and the changes I’ve made are backward compatible.

The QxCompiler fixes are to do with dependencies – basically, the load dependencies of a Qooxdoo app are greatly complicated because classes can have a defer() method, which allows code to be run before the app is fully loaded.  The way (I think/guess) that the generate.py does it is to recursively interpret the code in .defer() and all of the methods it calls, ie it tries to predict at compile-time what methods will be called at runtime so that it can make sure that the load order is correct.  As you can imagine this is non-trivial, but IMHO this makes it easy for minor code changes to have a big impact on load order and to cause unresolvable recursive dependency issues.

My solution is to not call the class .defer() method until all classes are loaded (that’s not strictly possible because some classes .defer() must be called, but the dependencies are a lot simpler).  This solution needs a a couple of backwards compatible mods, mostly in qx.Bootstrap, and these are in the qxcompiler branch


Cheers
John

From: Dietrich Streifert <[hidden email]>
Reply-To: qooxdoo Development <[hidden email]>
Date: Monday, 15 February 2016 at 08:04
To: qooxdoo Development <[hidden email]>
Subject: Re: [qooxdoo-devel] QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

Hi John

Congratulations!

One question regarding this passage in the README.md:

NOTE While QxCompiler is backward compatible with generate.py and does not require changes to your code, ES6 support requires that strict mode is enabled and this can introduce a few minor compatibility issues; also, QxCompiler uses a new trick for managing dependencies and for both of these reasons you must use my fork of Qooxdoo (https://github.com/johnspackman/qooxdoo and use the qxcompiler branch). My fork is based on Qooxdoo 5.0.1.

Could you explain which changes where necessary in qooxdoo to make it work with your compiler? Would it maybe be possible to "document" those changes via a pull request against qooxdoo/qooxdoo?

Another question is related to browser compatibilities: are they changed/raised because of the ES6 and strict mode changes? Will it still run on IE8?

Regards
Dietrich


------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________ qooxdoo-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
qooxdoo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

John Spackman
In reply to this post by John Spackman
HI Dimitri

Ready and waiting ;)

Re 1 testdata/qxt: fixed now, but what you did is fine (my version just has a couple of irrelevant test/demo tweaks)

Re 2 build process, messages: fixed

Re 2 build process, too many open files: ah – OK I can throttle that, I just saw that on my mac ulimit is unlimited (!)

Re 3 icon resource not found: when you copied the source directory into testdata/qxt did you copy the resource directory also?  I don’t understand this because it works for me :(  You’re right that it should be looking in the resource/ folder for resources, and that’s what mine’s doing here.  I’ve just done a fresh checkout from the repo and tried it OK, so please can you try pulling, deleting the build-output directory and try again?  

Re 3 all resources being copied: fixed (this should fix your ulimit issue too)

Re 4 UploadMgr: yes that’s exactly how to add a contrib.  I’ve fixed the problem in UploadMgr so that it is not strict mode compatible and added it to the qxt app as a demo.

Re 5 this.self() not working: fixed

Re 6 environment settings: yes, I’ve added a property to qxcompiler.Maker that is a map you can set; test/compile-app-demo.js now looks like this:
// Makers use an Analyser to figure out what the Target should write
var maker = new qxcompiler.makers.SimpleApp("qxt.Application", "qxt.theme.Theme").set({
// Targets know how to output an application
target: new qxcompiler.targets.SourceTarget("../testdata/qxt/source-output")
});
maker.setEnvironment({
"qxt.customEnvironment": "this is custom (source target)"
});
Re 7: Locales and translation: Locales are supported but translations not yet; it should be straightforward though, so I’ll take a look at translations maybe tonight, probably tomorrow

Re 8: Large build-target output: the output isn’t compressed/minified yet, but I’ll be adding that in “real soon now”; I’m planning on using UglifyJS to do this, my quick test so far shows that the size of the qxt app is reduced from 3.9Mb to 1.1Mb, so not as good yet as generate.py but in the general area.  I think the remaining reduction in size will come by dead code elimination because the superfluous qx.debug code  is not removed from the Build Target yet

Well I’m pleased with that for a morning’s work :)   There’s a bit more to do that I’ll get onto tonight/tomorrow but it should be enough to get you going on the next step.  Don’t forget to checkout the qooxdoo repo as well as the main QxCompiler one.

Cheers
John


From: Dimitri <[hidden email]>
Reply-To: qooxdoo Development <[hidden email]>
Date: Monday, 15 February 2016 at 03:28
To: qooxdoo Development <[hidden email]>
Subject: Re: [qooxdoo-devel] QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

Hi John! Ready for another dozen of questions and bug reports? ;)

1. As for testdata/qxt directory - yup, it is indeed present, but I don't see "source" subdirectory, where I would expect the actual application code to reside. So I just ran "create-application.py -n qxt" in another location and copied the generated "source" subdirectory to testdata/qxt. Is it how it is supposed to work?

2. The build process spits a lot of trace messages like this:
Function enter: anonymous
Function exit: anonymous

SourceTarget completes successfully, but BuildTarget fails with the following:

2016-02-15 04:10:22.045 [info ] makers          Writing target Build Target: ../testdata/qxt/build-output/
events.js:141
      throw er; // Unhandled 'error' event
      ^

Error: EMFILE: too many open files, open '../testdata/qxt/build-output//resource/qx/icon/Tango/48/actions/go-home.png'
    at Error (native)

I had to increase open file descriptors limit (ulimit -n 8192 worked for me). If this cannot be fixed/optimized, probably it should be reflected in the docs.

3. There seems to be a big mess with resources. Test app works well in its source variant, but the single-script build fails to load an icon:

000472 qx.ui.basic.Image[45-0]: Image could not be loaded: qxt/test.png boot.js:54698:13
000474 qx.ui.basic.Image[45-0]: Image could not be loaded: qxt/test.png boot.js:54698:13
000569 ImageLoader: Not recognized format of external image 'qxt/test.png'! boot.js:54698:13

Shouldn't it look in "resources" subdirectory instead? The subdir is present and does contain resources; however, it seems to contain *all* the available qooxdoo+app resources (~3600 files, ~25M). generate.py usually produces a ~150K subdir containing resources that are actually required.

At the same time, my application was unable to load resources neither in source nor in build variant. In both cases it tried to (unsuccessfully) look up resources in that very same "source-output" or "build-output" directory.

4. Our project uses your (excellent :) UploadMgr contrib. I've added the following to the async.series block:

        function(cb) {
          maker.addLibrary("/path/to/qooxdoo-contrib/UploadMgr", cb);
        },

Is this the right way to integrate a contrib into the project?

Can't yet say whether it worked or not, as I didn't manage to make application work (see below). I've noticed the following in console:
SyntaxError: in strict mode code, functions may be declared only at top level or immediately within another function (source-output/transpiled//com/zenesis/qx/upload/XhrHandler.js:212:19)
(original file line 212, function sendAsMime definition)

5. One of my classes invokes its static method from a constructor. According to qooxdoo docs, this should be done this way:
this.self(arguments).foo();

The application failed to load because of "this.constructor.self" being undefined. The following however worked fine:
this.constructor.foo();

This could potentially break applications that access static members from constructors and/or instance methods. (I'm not sure if constructor is some special case.)

6. Is there a way to pass qooxdoo environment keys to the app a la config.js? Much better if the values could be passed via command line. Our build system produces metadata (version, build number, revision, timestamp) which we are planning to pass to the app in the form of qx.core.Environment keys.

7. What about locales & translation? For us, it is essential because our app is already bilingual, and more languages are to come.

8. For our app, "generate.py build" produces a ~900K script, and it takes less than a second for the app to be up and running.
QxCompiler/BuildTarget produces a huge ~5M single script, and application startup takes up to 10 seconds (!)
What is the recommended way to produce optimized/minified build that would be comparable to the original one?

Okay, that's enough for today :) Tomorrow I'll try to figure out the cause of another failure that prevents our app from starting up. It's something related to JSON unmarshalling; the data is unmarshalled as auto-generated class (qx.data.model.prop1"prop2"prop3[123-0]), while it should be an application class resolved by a marshal delegate. This works fine with generate.py.

Good luck!
Dimitri

Further to this - I've added some docs about the API, but also seen that the BuildTarget has a bug and doesn't work - it's only a minor issue but I probably wont have time to fix it until tonight
 
John
 
 
 

From: "John Spackman" <[hidden email]>
Sent: Sunday, February 14, 2016 9:50 AM
To: "qooxdoo Development" <[hidden email]>
Subject: Re: [qooxdoo-devel] QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications
 
Hi Dimitri
 
Thanks for checking out so quickly :)
 
Re: npm and docs: done
 
Re: qooxdoo submodule and https: fixed
 
Re: QOOXDOO_PATH: no - that's a generator.py/config.json setting and config.json files are completely ignored; when using the QxCompiler API, once you have told it where the Qooxdoo library is you have done the equivalent of setting QOOXDOO_PATH.  Of course, as it's API based you can define variables etc as suits you best :)
 
In test/compile-app-demo.js it adds the Qooxdoo library like this:
    maker.addLibrary("../qooxdoo/framework", cb);
 
Just change the path to point to your preferred location for qooxdoo.  To make this a bit clearer, I've just modified the test/compile-app-demo.js to declare a variable QOOXDOO_PATH
 
I expect that the command line version will imply have a search path for libraries and just auto-discover by searching for Manifest.json files.
 
Re: missing testdata/qxt: something went wrong with your checkout, that directory is definitely there.
 
Re: build (and source-hybrid) targets: to control output for Source vs Build vs Source-Hybrid, the API classes qxcompiler.targets.SourceTarget, qxcompiler.targets.BuildTarget, and qxcompiler.targets.SourceHybridTarget are used; test/compile-app-demo.js uses the SourceTarget but you should be able to switch it easily enough.  I say "should" because I realise now that I havn't tested them for a few weeks now and I should give them the once over ASAP :)  I've got to go out shortly so I'll take a look this evening.
 
John
 
 

From: "Dimitri" <[hidden email]>
Sent: Sunday, February 14, 2016 2:27 AM
To: "qooxdoo Development" <[hidden email]>
Subject: Re: [qooxdoo-devel] QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications
 
John, congratulations with the long awaited release! :) qooxdoo guys, do you think that project like this could at some moment land in qooxdoo and get official support? What about qooxdoo patches (mostly strict mode compatibility related, AFAIK) - could they be merged upstream? This would reduce maintainship costs for those who want to experiment with both QxCompiler and official toolchain. I didn't yet try QxCompiler with my project - I feel I yet lack some understanding of how it works. However, I tried to test it with the skeleton app. Off the top of my head: - one needs to do "npm install" first and to run test scripts with "node <script>.js". This might be quite obvious for those experienced with Node, but I guess the docs will nevertheless benefit from mentioning this; - it's not easy to clone a repo unless you've set up Github SSH access. This is because of "qooxdoo" submodule pointing to "[hidden email]:john spackman/qooxdoo.git". Could it be a HTTPS URL instead? - does QxCompiler honour QOOXDOO_PATH setting? How do I define qooxdoo location if it is different from the bundled one? - testdata/qxt directory doesn't contain skeleton app. In order to play with test scripts, one needs to create the app manually (as "skeleton.Application"?) and copy it to the said location; - does lib/compiler.jar really belong there? Most importantly, I was unable to find clear instructions how to produce a single-script minified build (a-la "generate.py build"). By deafult, QxCompiler produces a bunch of JS files, which significantly slows down loading and, obviously, is not acceptable for production. Is it possible at all with the current version (maybe in combination with classic generator)? I hope these are just minor issues. After all, you've done a great job :) I wish you good luck and further progress wih QxCompiler. Dimitri > Hi all > > There is a first release of my QxCompiler that adds ES6 to Qooxdoo > applications and replaces the generate.py toolchain with a faster, > 100% Javascript tool that is easily extensible. > > You can find the first release at GitHub here: https://github.com/joh > nspackman/qxcompiler > > It’s an alpha release, but something that’s been in development for a > while now and which I’m starting to build into my production > servers.   > > I’m very open to pull requests or collaboration, and keen to see this > become a useful tool for myself as well as others so any questions > etc please ask. > > Regards > John > > > > > > ------------------------------------------------------------------- > ----------- > Site24x7 APM Insight: Get Deep Visibility into Application > Performance > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month > Monitor end-to-end web transactions and take corrective actions now > Troubleshoot faster and improve end-user experience. Signup Now! > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > _______________________________________________ > qooxdoo-devel mailing list > [hidden email] > https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 _______________________________________________ qooxdoo-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
qooxdoo-devel mailing list
[hidden email]https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________ qooxdoo-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
qooxdoo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

Defero
In reply to this post by John Spackman-3
Hi,

congratulations on the release

I've already started to play around a bit and will send you a pull request for it. While trying to set it up, i've run into a little snag at the following:

qxcompiler\lib\util.js
line: 305
code : return fs.mkdir(made, function(err) {

and because fs.mkdir (in my version of node at least) doesnt create folders recursively, it failed the compilation of the compile-app-demo.js. I changed to the mkdirp module.

this is looking sweet though :)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

mitya
In reply to this post by John Spackman
Hi! Tried again - things are much better now :) However, two issues still remain, one of them being resource management.

Indeed, test app loads its icons correctly in both versions. Unfortunately, this is not the case for my app. What I've found yet:

- the app loads all the qx.* resources correctly (window minimize/maximize/close icons etc.);
- none of the application's own resources could be loaded. The request is made to "build-output/myapp/foo.png", which obviously results in a 404.
- in resource-db.json, all resources that *do* load contain non-empty metadata block, while those that do *not* have an empty literal {}. The same could be observed in qxt for com/zenesis/qx/upload/* resources.

(Please note that our app contains not only images, but audio files in different formats, mainly mp3/ogg. We put them into "resource/sounds".)

The app still cannot load. That unmarshalling error seems to have gone, but now I've got a "too much recursion" somewhere around the same location in code. (stacktrace)
I'll try to produce MWEs for both issues later today. Some other stuff to consider:

1. NodeJS <= 0.12 is not supported, while 4.x and 5.x are fine (due to object literal extension used at lib/qxcompiler/ClassFile.js:329). Some distros still ship NodeJS 0.x with their stable versions. I think this should be mentioned in the docs. OTOH, why not babelize QxCompiler itself? :)
2. What practice would you recommend for using QxCompiler on a build server? For each build, a clean checkout is made. Doing npm install in the project directory is very expensive (~15MB deps). Assuming QxCompiler resides in its own directory, what would you recommend? symlink <project>/node_modules -> <QxCompiler>/node_modules? install deps with npm -g and use <a href="https://linklocal? ">linklocal ? Ideally, I'd like some practice that wouldn't change when we finally have a NPM distro of QxCompiler.
3. It's worth mentioning in the docs that ImageMagick is required (due to "identify" binary being used in qxcompiler/ResourceManager.js).

Cheers,
Dimitri

HI Dimitri

Ready and waiting ;)

Re 1 testdata/qxt: fixed now, but what you did is fine (my version just has a couple of irrelevant test/demo tweaks)

Re 2 build process, messages: fixed

Re 2 build process, too many open files: ah – OK I can throttle that, I just saw that on my mac ulimit is unlimited (!)

Re 3 icon resource not found: when you copied the source directory into testdata/qxt did you copy the resource directory also?  I don’t understand this because it works for me :(  You’re right that it should be looking in the resource/ folder for resources, and that’s what mine’s doing here.  I’ve just done a fresh checkout from the repo and tried it OK, so please can you try pulling, deleting the build-output directory and try again?  

Re 3 all resources being copied: fixed (this should fix your ulimit issue too)

Re 4 UploadMgr: yes that’s exactly how to add a contrib.  I’ve fixed the problem in UploadMgr so that it is not strict mode compatible and added it to the qxt app as a demo.

Re 5 this.self() not working: fixed

Re 6 environment settings: yes, I’ve added a property to qxcompiler.Maker that is a map you can set; test/compile-app-demo.js now looks like this:
// Makers use an Analyser to figure out what the Target should write
var maker = new qxcompiler.makers.SimpleApp("qxt.Application", "qxt.theme.Theme").set({
// Targets know how to output an application
target: new qxcompiler.targets.SourceTarget("../testdata/qxt/source-output")
});
maker.setEnvironment({
"qxt.customEnvironment": "this is custom (source target)"
});
Re 7: Locales and translation: Locales are supported but translations not yet; it should be straightforward though, so I’ll take a look at translations maybe tonight, probably tomorrow

Re 8: Large build-target output: the output isn’t compressed/minified yet, but I’ll be adding that in “real soon now”; I’m planning on using UglifyJS to do this, my quick test so far shows that the size of the qxt app is reduced from 3.9Mb to 1.1Mb, so not as good yet as generate.py but in the general area.  I think the remaining reduction in size will come by dead code elimination because the superfluous qx.debug code  is not removed from the Build Target yet

Well I’m pleased with that for a morning’s work :)   There’s a bit more to do that I’ll get onto tonight/tomorrow but it should be enough to get you going on the next step.  Don’t forget to checkout the qooxdoo repo as well as the main QxCompiler one.

Cheers
John


From: Dimitri <[hidden email]>
Reply-To: qooxdoo Development <[hidden email]>
Date: Monday, 15 February 2016 at 03:28
To: qooxdoo Development <[hidden email]>
Subject: Re: [qooxdoo-devel] QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

Hi John! Ready for another dozen of questions and bug reports? ;)

1. As for testdata/qxt directory - yup, it is indeed present, but I don't see "source" subdirectory, where I would expect the actual application code to reside. So I just ran "create-application.py -n qxt" in another location and copied the generated "source" subdirectory to testdata/qxt. Is it how it is supposed to work?

2. The build process spits a lot of trace messages like this:
Function enter: anonymous
Function exit: anonymous

SourceTarget completes successfully, but BuildTarget fails with the following:

2016-02-15 04:10:22.045 [info ] makers          Writing target Build Target: ../testdata/qxt/build-output/
events.js:141
      throw er; // Unhandled 'error' event
      ^

Error: EMFILE: too many open files, open '../testdata/qxt/build-output//resource/qx/icon/Tango/48/actions/go-home.png'
    at Error (native)

I had to increase open file descriptors limit (ulimit -n 8192 worked for me). If this cannot be fixed/optimized, probably it should be reflected in the docs.

3. There seems to be a big mess with resources. Test app works well in its source variant, but the single-script build fails to load an icon:

000472 qx.ui.basic.Image[45-0]: Image could not be loaded: qxt/test.png boot.js:54698:13
000474 qx.ui.basic.Image[45-0]: Image could not be loaded: qxt/test.png boot.js:54698:13
000569 ImageLoader: Not recognized format of external image 'qxt/test.png'! boot.js:54698:13

Shouldn't it look in "resources" subdirectory instead? The subdir is present and does contain resources; however, it seems to contain *all* the available qooxdoo+app resources (~3600 files, ~25M). generate.py usually produces a ~150K subdir containing resources that are actually required.

At the same time, my application was unable to load resources neither in source nor in build variant. In both cases it tried to (unsuccessfully) look up resources in that very same "source-output" or "build-output" directory.

4. Our project uses your (excellent :) UploadMgr contrib. I've added the following to the async.series block:

        function(cb) {
          maker.addLibrary("/path/to/qooxdoo-contrib/UploadMgr", cb);
        },

Is this the right way to integrate a contrib into the project?

Can't yet say whether it worked or not, as I didn't manage to make application work (see below). I've noticed the following in console:
SyntaxError: in strict mode code, functions may be declared only at top level or immediately within another function (source-output/transpiled//com/zenesis/qx/upload/XhrHandler.js:212:19)
(original file line 212, function sendAsMime definition)

5. One of my classes invokes its static method from a constructor. According to qooxdoo docs, this should be done this way:
this.self(arguments).foo();

The application failed to load because of "this.constructor.self" being undefined. The following however worked fine:
this.constructor.foo();

This could potentially break applications that access static members from constructors and/or instance methods. (I'm not sure if constructor is some special case.)

6. Is there a way to pass qooxdoo environment keys to the app a la config.js? Much better if the values could be passed via command line. Our build system produces metadata (version, build number, revision, timestamp) which we are planning to pass to the app in the form of qx.core.Environment keys.

7. What about locales & translation? For us, it is essential because our app is already bilingual, and more languages are to come.

8. For our app, "generate.py build" produces a ~900K script, and it takes less than a second for the app to be up and running.
QxCompiler/BuildTarget produces a huge ~5M single script, and application startup takes up to 10 seconds (!)
What is the recommended way to produce optimized/minified build that would be comparable to the original one?

Okay, that's enough for today :) Tomorrow I'll try to figure out the cause of another failure that prevents our app from starting up. It's something related to JSON unmarshalling; the data is unmarshalled as auto-generated class (qx.data.model.prop1"prop2"prop3[123-0]), while it should be an application class resolved by a marshal delegate. This works fine with generate.py.

Good luck!
Dimitri

Further to this - I've added some docs about the API, but also seen that the BuildTarget has a bug and doesn't work - it's only a minor issue but I probably wont have time to fix it until tonight
 
John
 
 
 

From: "John Spackman" <[hidden email]>
Sent: Sunday, February 14, 2016 9:50 AM
To: "qooxdoo Development" <[hidden email]>
Subject: Re: [qooxdoo-devel] QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications
 
Hi Dimitri
 
Thanks for checking out so quickly :)
 
Re: npm and docs: done
 
Re: qooxdoo submodule and https: fixed
 
Re: QOOXDOO_PATH: no - that's a generator.py/config.json setting and config.json files are completely ignored; when using the QxCompiler API, once you have told it where the Qooxdoo library is you have done the equivalent of setting QOOXDOO_PATH.  Of course, as it's API based you can define variables etc as suits you best :)
 
In test/compile-app-demo.js it adds the Qooxdoo library like this:
    maker.addLibrary("../qooxdoo/framework", cb);
 
Just change the path to point to your preferred location for qooxdoo.  To make this a bit clearer, I've just modified the test/compile-app-demo.js to declare a variable QOOXDOO_PATH
 
I expect that the command line version will imply have a search path for libraries and just auto-discover by searching for Manifest.json files.
 
Re: missing testdata/qxt: something went wrong with your checkout, that directory is definitely there.
 
Re: build (and source-hybrid) targets: to control output for Source vs Build vs Source-Hybrid, the API classes qxcompiler.targets.SourceTarget, qxcompiler.targets.BuildTarget, and qxcompiler.targets.SourceHybridTarget are used; test/compile-app-demo.js uses the SourceTarget but you should be able to switch it easily enough.  I say "should" because I realise now that I havn't tested them for a few weeks now and I should give them the once over ASAP :)  I've got to go out shortly so I'll take a look this evening.
 
John
 
 

From: "Dimitri" <[hidden email]>
Sent: Sunday, February 14, 2016 2:27 AM
To: "qooxdoo Development" <[hidden email]>
Subject: Re: [qooxdoo-devel] QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications
 
John, congratulations with the long awaited release! :) qooxdoo guys, do you think that project like this could at some moment land in qooxdoo and get official support? What about qooxdoo patches (mostly strict mode compatibility related, AFAIK) - could they be merged upstream? This would reduce maintainship costs for those who want to experiment with both QxCompiler and official toolchain. I didn't yet try QxCompiler with my project - I feel I yet lack some understanding of how it works. However, I tried to test it with the skeleton app. Off the top of my head: - one needs to do "npm install" first and to run test scripts with "node <script>.js". This might be quite obvious for those experienced with Node, but I guess the docs will nevertheless benefit from mentioning this; - it's not easy to clone a repo unless you've set up Github SSH access. This is because of "qooxdoo" submodule pointing to "[hidden email]:john spackman/qooxdoo.git". Could it be a HTTPS URL instead? - does QxCompiler honour QOOXDOO_PATH setting? How do I define qooxdoo location if it is different from the bundled one? - testdata/qxt directory doesn't contain skeleton app. In order to play with test scripts, one needs to create the app manually (as "skeleton.Application"?) and copy it to the said location; - does lib/compiler.jar really belong there? Most importantly, I was unable to find clear instructions how to produce a single-script minified build (a-la "generate.py build"). By deafult, QxCompiler produces a bunch of JS files, which significantly slows down loading and, obviously, is not acceptable for production. Is it possible at all with the current version (maybe in combination with classic generator)? I hope these are just minor issues. After all, you've done a great job :) I wish you good luck and further progress wih QxCompiler. Dimitri > Hi all > > There is a first release of my QxCompiler that adds ES6 to Qooxdoo > applications and replaces the generate.py toolchain with a faster, > 100% Javascript tool that is easily extensible. > > You can find the first release at GitHub here: https://github.com/joh > nspackman/qxcompiler > > It’s an alpha release, but something that’s been in development for a > while now and which I’m starting to build into my production > servers.   > > I’m very open to pull requests or collaboration, and keen to see this > become a useful tool for myself as well as others so any questions > etc please ask. > > Regards > John > > > > > > ------------------------------------------------------------------- > ----------- > Site24x7 APM Insight: Get Deep Visibility into Application > Performance > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month > Monitor end-to-end web transactions and take corrective actions now > Troubleshoot faster and improve end-user experience. Signup Now! > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > _______________________________________________ > qooxdoo-devel mailing list > [hidden email] > https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 _______________________________________________ qooxdoo-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
qooxdoo-devel mailing list
[hidden email]https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________ qooxdoo-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
qooxdoo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
qooxdoo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

thron7
In reply to this post by John Spackman-3
John,

this looks interesting! I like the API-based approach (Reminds me of the Clojure "boot" build system's tag line, "Builds are programs"[1]). It seems you have been working on that for a while.

[1] http://boot-clj.com/

On Mon, Feb 15, 2016 at 9:28 AM, John Spackman <[hidden email]> wrote:


The QxCompiler fixes are to do with dependencies – basically, the load dependencies of a Qooxdoo app are greatly complicated because classes can have a defer() method, which allows code to be run before the app is fully loaded.  The way (I think/guess) that the generate.py does it is to recursively interpret the code in .defer() and all of the methods it calls, ie it tries to predict at compile-time what methods will be called at runtime so that it can make sure that the load order is correct.  As you can imagine this is non-trivial, but IMHO this makes it easy for minor code changes to have a big impact on load order and to cause unresolvable recursive dependency issues.

My solution is to not call the class .defer() method until all classes are loaded (that’s not strictly possible because some classes .defer() must be called, but the dependencies are a lot simpler).  This solution needs a a couple of backwards compatible mods, mostly in qx.Bootstrap, and these are in the qxcompiler branch


What really complicates dependency inference is to find the transitive closure for each (what you put as "recursively interpret the code"). But I think you need that for both load-time and run-time dependencies alike. This entails that any small change in far away code can have an impact on the overall set of classes and their load order. (But only load-time makes cyclic dependencies an issue, and that's probably what you care about).

Once recursive analysis is in place, adding the symbols from .defer() to the load-time dependencies is easy as you write. So it's not that .defer() makes the general mechanism of treating dependencies more difficult - it just enlarges the set of load-time dependencies, and hence increases the risk of not being able to create a partial order for all the classes. But .defer() is not particularly more problematic than, say, static initializers in the class, or #require's.

But I see how .defer() is a good target to minimize on that risk.

T.

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
qooxdoo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

John Spackman
Thanks Thomas :)

I like the API approach too, I think it opens up some possibilities (I remember you had it on your todo list for a while and I can see why).  Its definitely been a while coming, I wrote the proof of concept years ago with Esprima but mortgage-paying work always took priority!

With my approach to dependencies, QxCompiler is taking a shortcut but reducing that target appears to be very profitable; there have been a couple of cases where I’ve had to add in @require to Qooxdoo classes, this is typically where (e.g.) a qx.core.Environment provider class uses a static method to initialise instead of directly in .defer() and there is an additional dependency, but there’s a lot of cases where explicit @require() is already present so my mods to the framework have been kept to a minimum.

I quite like that the database (the equivalent of generator’s cache) is kept quite small too - around 1Mb, and tracking the dependencies of methods would add a lot of data as well as code complexity so if I can keep it this way then that would be ideal.

Regards
John

From: thron7 <[hidden email]>
Reply-To: qooxdoo Development <[hidden email]>
Date: Monday, 15 February 2016 at 22:34
To: qooxdoo Development <[hidden email]>
Subject: Re: [qooxdoo-devel] QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

John,

this looks interesting! I like the API-based approach (Reminds me of the Clojure "boot" build system's tag line, "Builds are programs"[1]). It seems you have been working on that for a while.

[1] http://boot-clj.com/

On Mon, Feb 15, 2016 at 9:28 AM, John Spackman <[hidden email]> wrote:


The QxCompiler fixes are to do with dependencies – basically, the load dependencies of a Qooxdoo app are greatly complicated because classes can have a defer() method, which allows code to be run before the app is fully loaded.  The way (I think/guess) that the generate.py does it is to recursively interpret the code in .defer() and all of the methods it calls, ie it tries to predict at compile-time what methods will be called at runtime so that it can make sure that the load order is correct.  As you can imagine this is non-trivial, but IMHO this makes it easy for minor code changes to have a big impact on load order and to cause unresolvable recursive dependency issues.

My solution is to not call the class .defer() method until all classes are loaded (that’s not strictly possible because some classes .defer() must be called, but the dependencies are a lot simpler).  This solution needs a a couple of backwards compatible mods, mostly in qx.Bootstrap, and these are in the qxcompiler branch


What really complicates dependency inference is to find the transitive closure for each (what you put as "recursively interpret the code"). But I think you need that for both load-time and run-time dependencies alike. This entails that any small change in far away code can have an impact on the overall set of classes and their load order. (But only load-time makes cyclic dependencies an issue, and that's probably what you care about).

Once recursive analysis is in place, adding the symbols from .defer() to the load-time dependencies is easy as you write. So it's not that .defer() makes the general mechanism of treating dependencies more difficult - it just enlarges the set of load-time dependencies, and hence increases the risk of not being able to create a partial order for all the classes. But .defer() is not particularly more problematic than, say, static initializers in the class, or #require's.

But I see how .defer() is a good target to minimize on that risk.

T.
------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________ qooxdoo-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
qooxdoo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

John Spackman
In reply to this post by Defero
Hi Defero

Thanks :)

I found a bug in util.mkpath yesterday that might have been the problem you found - mkpath would start to create the folders and then stop, but that’s fixed now.

Are you thinking of having a look at part loading?

John




On 15/02/2016, 17:15, "Defero" <[hidden email]> wrote:

>Hi,
>
>congratulations on the release
>
>I've already started to play around a bit and will send you a pull request
>for it. While trying to set it up, i've run into a little snag at the
>following:
>
>qxcompiler\lib\util.js
>line: 305
>code : return fs.mkdir(made, function(err) {
>
>and because fs.mkdir (in my version of node at least) doesnt create folders
>recursively, it failed the compilation of the compile-app-demo.js. I changed
>to the mkdirp module.
>
>this is looking sweet though :)
>
>
>
>--
>View this message in context: http://qooxdoo.678.n2.nabble.com/QxCompiler-add-ES6-faster-compilation-and-100-Javascript-API-to-building-applications-tp7587992p7588003.html
>Sent from the qooxdoo mailing list archive at Nabble.com.
>
>------------------------------------------------------------------------------
>Site24x7 APM Insight: Get Deep Visibility into Application Performance
>APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>Monitor end-to-end web transactions and take corrective actions now
>Troubleshoot faster and improve end-user experience. Signup Now!
>http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
>_______________________________________________
>qooxdoo-devel mailing list
>[hidden email]
>https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel



------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
qooxdoo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

John Spackman
In reply to this post by mitya
Hi Dimitri

So it sounds like resources with paths have an issue, or maybe libraries with “.” in the namespace – I’ll have a look this morning but as it’s happening with UploadMgr then I have a reproducible case I can work on.

I’ve had a look at the stack trace – but I can’t see any clues to what might be causing it; by marshalling, do you mean the qx.data.marshall.*?  That should not be affected, the only change to code was to allow .defer() to not be called immediately after the class was defined.  I’ve just realised that this means that dynamically defined classes will not have their defer called, and there’s a fix in the next commit for that but I don’t think that’s the issue your coming across.

Re Node 0.12: I think there was another problem with early versions of Node <= 0.12 other than just language – I can’t remember exactly what now but there was something that didn’t work properly until I upgraded.  But you’re right, QxCompiler could be babelified :)  At one point, the Babel plugin part was written as a proper, standalone babel plugin in ES6 but I had some trouble getting Webstorm to compile and debug properly so I switched back to native code

Re build server: I’d expect QxCompiler to be installed once, globally and then used externally to the projects, i.e. your build code could do the project checkout and then use the API to build each project.  The project wouldn’t know about QxCompiler at all.

This sounds similar to my use case, except my builds are on-the-fly on product servers rather than as part of release: In my case I have multi homed Tomcat servers, and each website uses Qooxdoo on the server and client; the server apps generate html (think ASP/PHP/JSP but with javascript & Qooxdoo running under Rhino) as well as implement server processes, and there is often more than one client application per website.  Doing a major release requires deleting and recompiling all applications (and typically the cache also), and with a minimum of 2 apps (1 server + 1 client) per website that can be quite expensive.  At the moment, making a minor patch to the presentation of the website (e.g. modifying the html output from my PHP/ASP-like pages) means an expensive build process too which is pretty tedious.  

So what I want is to have QxCompiler sitting as a background process, watching all these applications and the source files they depend on; when a source file changes it rebuilds the app instantly (e.g. 250ms per affected application) and notifies Tomcat to reload the server code if necessary.  At the moment, incremental builds take QxCompiler approx 700ms including the (not insignificant) overhead of starting up node.

Re requirements: I’ve updated the docs to match

I’m working on QxCompiler again this morning and should have an update for you by lunchtime 

Cheers
John

From: Dimitri <[hidden email]>
Reply-To: qooxdoo Development <[hidden email]>
Date: Monday, 15 February 2016 at 18:16
To: <[hidden email]>
Subject: Re: [qooxdoo-devel] QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

Hi! Tried again - things are much better now :) However, two issues still remain, one of them being resource management.

Indeed, test app loads its icons correctly in both versions. Unfortunately, this is not the case for my app. What I've found yet:

- the app loads all the qx.* resources correctly (window minimize/maximize/close icons etc.);
- none of the application's own resources could be loaded. The request is made to "build-output/myapp/foo.png", which obviously results in a 404.
- in resource-db.json, all resources that *do* load contain non-empty metadata block, while those that do *not* have an empty literal {}. The same could be observed in qxt for com/zenesis/qx/upload/* resources.

(Please note that our app contains not only images, but audio files in different formats, mainly mp3/ogg. We put them into "resource/sounds".)

The app still cannot load. That unmarshalling error seems to have gone, but now I've got a "too much recursion" somewhere around the same location in code. (stacktrace)
I'll try to produce MWEs for both issues later today. Some other stuff to consider:

1. NodeJS <= 0.12 is not supported, while 4.x and 5.x are fine (due to object literal extension used at lib/qxcompiler/ClassFile.js:329). Some distros still ship NodeJS 0.x with their stable versions. I think this should be mentioned in the docs. OTOH, why not babelize QxCompiler itself? :)
2. What practice would you recommend for using QxCompiler on a build server? For each build, a clean checkout is made. Doing npm install in the project directory is very expensive (~15MB deps). Assuming QxCompiler resides in its own directory, what would you recommend? symlink <project>/node_modules -> <QxCompiler>/node_modules? install deps with npm -g and use <a href="https://linklocal? ">linklocal ? Ideally, I'd like some practice that wouldn't change when we finally have a NPM distro of QxCompiler.
3. It's worth mentioning in the docs that ImageMagick is required (due to "identify" binary being used in qxcompiler/ResourceManager.js).

Cheers,
Dimitri


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
qooxdoo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

Dietrich Streifert
Hi John,

we are using Centos 7.2 aka RHEL 7.2 which delivers nodejs 0.10.36
through epel repository.

Regards
Dietrich


Am 16.02.2016 um 08:26 schrieb John Spackman:
> Re Node 0.12: I think there was another problem with early versions of
> Node <= 0.12 other than just language – I can’t remember exactly what
> now but there was something that didn’t work properly until I
> upgraded.  But you’re right, QxCompiler could be babelified :)  At one
> point, the Babel plugin part was written as a proper, standalone babel
> plugin in ES6 but I had some trouble getting Webstorm to compile and
> debug properly so I switched back to native code
>


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
qooxdoo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

Defero
In reply to this post by John Spackman
Hi,

currently i'm still in the process of playing around with it, looking under the hood a bit in my free time But yeah, it will be on my priority todo list to create a part loader.


Defero
dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

dev
Congratulations John!

This is the major step taken the last 18 months!!! (cause nothing much happens with qooxdoo anymore while ExtJS and other frameworks develop quite fast)
You have proven it come be done in a very delicate way.
You give the core team a huge challenge...the question is if they can handle it the right way. No proven good track record in this area...;-(
Anyways, we have been testing it and it looks amazing and I am sure all your work can be reused by all of us to increase modularity...

Thanks!

Stefan

> Thanks Thomas :)
>
> I like the API approach too, I think it opens up some possibilities (I remember you had it on your todo list for a while and I can see why).  Its definitely been a while coming, I wrote the proof of concept years ago with Esprima but mortgage-paying work always took priority!
>
> With my approach to dependencies, QxCompiler is taking a shortcut but reducing that target appears to be very profitable; there have been a couple of cases where I’ve had to add in @require to Qooxdoo classes, this is typically where (e.g.) a qx.core.Environment provider class uses a static method to initialise instead of directly in .defer() and there is an additional dependency, but there’s a lot of cases where explicit @require() is already present so my mods to the framework have been kept to a minimum.
>
> I quite like that the database (the equivalent of generator’s cache) is kept quite small too - around 1Mb, and tracking the dependencies of methods would add a lot of data as well as code complexity so if I can keep it this way then that would be ideal.
>
> Regards
> John
>
> From:  thron7 <[hidden email]>
> Reply-To:  qooxdoo Development <[hidden email]>
> Date:  Monday, 15 February 2016 at 22:34
> To:  qooxdoo Development <[hidden email]>
> Subject:  Re: [qooxdoo-devel] QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications
>
> John,
>
> this looks interesting! I like the API-based approach (Reminds me of the Clojure "boot" build system's tag line, "Builds are programs"[1]). It seems you have been working on that for a while.
>
> [1] http://boot-clj.com/
>
> On Mon, Feb 15, 2016 at 9:28 AM, John Spackman <[hidden email]> wrote:
>
>
> The QxCompiler fixes are to do with dependencies – basically, the load dependencies of a Qooxdoo app are greatly complicated because classes can have a defer() method, which allows code to be run before the app is fully loaded.  The way (I think/guess) that the generate.py does it is to recursively interpret the code in .defer() and all of the methods it calls, ie it tries to predict at compile-time what methods will be called at runtime so that it can make sure that the load order is correct.  As you can imagine this is non-trivial, but IMHO this makes it easy for minor code changes to have a big impact on load order and to cause unresolvable recursive dependency issues.
>
> My solution is to not call the class .defer() method until all classes are loaded (that’s not strictly possible because some classes .defer() must be called, but the dependencies are a lot simpler).  This solution needs a a couple of backwards compatible mods, mostly in qx.Bootstrap, and these are in the qxcompiler branch
>
> There’s bit more detail here: https://github.com/johnspackman/qxcompiler/blob/master/docs/Dependencies.md
>
> What really complicates dependency inference is to find the transitive closure for each (what you put as "recursively interpret the code"). But I think you need that for both load-time and run-time dependencies alike. This entails that any small change in far away code can have an impact on the overall set of classes and their load order. (But only load-time makes cyclic dependencies an issue, and that's probably what you care about).
>
> Once recursive analysis is in place, adding the symbols from .defer() to the load-time dependencies is easy as you write. So it's not that .defer() makes the general mechanism of treating dependencies more difficult - it just enlarges the set of load-time dependencies, and hence increases the risk of not being able to create a partial order for all the classes. But .defer() is not particularly more problematic than, say, static initializers in the class, or #require's.
>
> But I see how .defer() is a good target to minimize on that risk.
>
> T.
> ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________ qooxdoo-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel


> ------------------------------------------------------------------------------
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140


> _______________________________________________
> qooxdoo-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel



------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
qooxdoo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

John Spackman
In reply to this post by Dietrich Streifert
Hi Dietrich

Yes, I see your point - I use Centos 6.x so I have the same issue.  When the core functionality is settled, I’ll go back and look again at using Babel and see what version of Node we can support

Regards
John





On 16/02/2016, 08:18, "Dietrich Streifert" <[hidden email]> wrote:

>Hi John,
>
>we are using Centos 7.2 aka RHEL 7.2 which delivers nodejs 0.10.36
>through epel repository.
>
>Regards
>Dietrich
>
>
>Am 16.02.2016 um 08:26 schrieb John Spackman:
>> Re Node 0.12: I think there was another problem with early versions of
>> Node <= 0.12 other than just language – I can’t remember exactly what
>> now but there was something that didn’t work properly until I
>> upgraded.  But you’re right, QxCompiler could be babelified :)  At one
>> point, the Babel plugin part was written as a proper, standalone babel
>> plugin in ES6 but I had some trouble getting Webstorm to compile and
>> debug properly so I switched back to native code
>>
>
>
>------------------------------------------------------------------------------
>Site24x7 APM Insight: Get Deep Visibility into Application Performance
>APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>Monitor end-to-end web transactions and take corrective actions now
>Troubleshoot faster and improve end-user experience. Signup Now!
>http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
>_______________________________________________
>qooxdoo-devel mailing list
>[hidden email]
>https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel



------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
qooxdoo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

John Spackman
In reply to this post by dev
Hi Stefan

Glad you like it :)  I think that the important thing about a VCS like git is that it is easier to take things forward ourselves if we want to.  So my qxcompiler branch is designed to easily fit in with anyone elses.  1&1 have their own corporate direction which may not have much focus on Qooxdoo any more, and although there might be a connection between the MIT license and decentralising the core repo I haven’t heard anything yet

But I’d be interested to collaborate on an official community fork; there may not be a huge explosion of development activity ;) but it would give us (the community) the opportunity to take the project forward, merging in changes by the core 1&1 team.

What do you/anyone think?

John




On 16/02/2016, 11:25, "[hidden email]" <[hidden email]> wrote:

>Congratulations John!
>
>This is the major step taken the last 18 months!!! (cause nothing much happens with qooxdoo anymore while ExtJS and other frameworks develop quite fast)
>You have proven it come be done in a very delicate way.
>You give the core team a huge challenge...the question is if they can handle it the right way. No proven good track record in this area...;-(
>Anyways, we have been testing it and it looks amazing and I am sure all your work can be reused by all of us to increase modularity...
>
>Thanks!
>
>Stefan
>
>> Thanks Thomas :)
>>
>> I like the API approach too, I think it opens up some possibilities (I remember you had it on your todo list for a while and I can see why).  Its definitely been a while coming, I wrote the proof of concept years ago with Esprima but mortgage-paying work always took priority!
>>
>> With my approach to dependencies, QxCompiler is taking a shortcut but reducing that target appears to be very profitable; there have been a couple of cases where I’ve had to add in @require to Qooxdoo classes, this is typically where (e.g.) a qx.core.Environment provider class uses a static method to initialise instead of directly in .defer() and there is an additional dependency, but there’s a lot of cases where explicit @require() is already present so my mods to the framework have been kept to a minimum.
>>
>> I quite like that the database (the equivalent of generator’s cache) is kept quite small too - around 1Mb, and tracking the dependencies of methods would add a lot of data as well as code complexity so if I can keep it this way then that would be ideal.
>>
>> Regards
>> John
>>
>> From:  thron7 <[hidden email]>
>> Reply-To:  qooxdoo Development <[hidden email]>
>> Date:  Monday, 15 February 2016 at 22:34
>> To:  qooxdoo Development <[hidden email]>
>> Subject:  Re: [qooxdoo-devel] QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications
>>
>> John,
>>
>> this looks interesting! I like the API-based approach (Reminds me of the Clojure "boot" build system's tag line, "Builds are programs"[1]). It seems you have been working on that for a while.
>>
>> [1] http://boot-clj.com/
>>
>> On Mon, Feb 15, 2016 at 9:28 AM, John Spackman <[hidden email]> wrote:
>>
>>
>> The QxCompiler fixes are to do with dependencies – basically, the load dependencies of a Qooxdoo app are greatly complicated because classes can have a defer() method, which allows code to be run before the app is fully loaded.  The way (I think/guess) that the generate.py does it is to recursively interpret the code in .defer() and all of the methods it calls, ie it tries to predict at compile-time what methods will be called at runtime so that it can make sure that the load order is correct.  As you can imagine this is non-trivial, but IMHO this makes it easy for minor code changes to have a big impact on load order and to cause unresolvable recursive dependency issues.
>>
>> My solution is to not call the class .defer() method until all classes are loaded (that’s not strictly possible because some classes .defer() must be called, but the dependencies are a lot simpler).  This solution needs a a couple of backwards compatible mods, mostly in qx.Bootstrap, and these are in the qxcompiler branch
>>
>> There’s bit more detail here: https://github.com/johnspackman/qxcompiler/blob/master/docs/Dependencies.md
>>
>> What really complicates dependency inference is to find the transitive closure for each (what you put as "recursively interpret the code"). But I think you need that for both load-time and run-time dependencies alike. This entails that any small change in far away code can have an impact on the overall set of classes and their load order. (But only load-time makes cyclic dependencies an issue, and that's probably what you care about).
>>
>> Once recursive analysis is in place, adding the symbols from .defer() to the load-time dependencies is easy as you write. So it's not that .defer() makes the general mechanism of treating dependencies more difficult - it just enlarges the set of load-time dependencies, and hence increases the risk of not being able to create a partial order for all the classes. But .defer() is not particularly more problematic than, say, static initializers in the class, or #require's.
>>
>> But I see how .defer() is a good target to minimize on that risk.
>>
>> T.
>> ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________ qooxdoo-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
>
>
>> ------------------------------------------------------------------------------
>> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>> Monitor end-to-end web transactions and take corrective actions now
>> Troubleshoot faster and improve end-user experience. Signup Now!
>> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
>
>
>> _______________________________________________
>> qooxdoo-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
>
>
>
>------------------------------------------------------------------------------
>Site24x7 APM Insight: Get Deep Visibility into Application Performance
>APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>Monitor end-to-end web transactions and take corrective actions now
>Troubleshoot faster and improve end-user experience. Signup Now!
>http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
>_______________________________________________
>qooxdoo-devel mailing list
>[hidden email]
>https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel



------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
qooxdoo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

John Spackman
In reply to this post by John Spackman
Hi Dimitri

So further to this – I’ve made a lot of progress with translations today, but I’ve run out of time so rather than release anything half baked I’ll hold off until tomorrow

John

From: John Spackman <[hidden email]>
Reply-To: qooxdoo Development <[hidden email]>
Date: Tuesday, 16 February 2016 at 07:26
To: qooxdoo Development <[hidden email]>
Subject: Re: [qooxdoo-devel] QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

Hi Dimitri

So it sounds like resources with paths have an issue, or maybe libraries with “.” in the namespace – I’ll have a look this morning but as it’s happening with UploadMgr then I have a reproducible case I can work on.

I’ve had a look at the stack trace – but I can’t see any clues to what might be causing it; by marshalling, do you mean the qx.data.marshall.*?  That should not be affected, the only change to code was to allow .defer() to not be called immediately after the class was defined.  I’ve just realised that this means that dynamically defined classes will not have their defer called, and there’s a fix in the next commit for that but I don’t think that’s the issue your coming across.

Re Node 0.12: I think there was another problem with early versions of Node <= 0.12 other than just language – I can’t remember exactly what now but there was something that didn’t work properly until I upgraded.  But you’re right, QxCompiler could be babelified :)  At one point, the Babel plugin part was written as a proper, standalone babel plugin in ES6 but I had some trouble getting Webstorm to compile and debug properly so I switched back to native code

Re build server: I’d expect QxCompiler to be installed once, globally and then used externally to the projects, i.e. your build code could do the project checkout and then use the API to build each project.  The project wouldn’t know about QxCompiler at all.

This sounds similar to my use case, except my builds are on-the-fly on product servers rather than as part of release: In my case I have multi homed Tomcat servers, and each website uses Qooxdoo on the server and client; the server apps generate html (think ASP/PHP/JSP but with javascript & Qooxdoo running under Rhino) as well as implement server processes, and there is often more than one client application per website.  Doing a major release requires deleting and recompiling all applications (and typically the cache also), and with a minimum of 2 apps (1 server + 1 client) per website that can be quite expensive.  At the moment, making a minor patch to the presentation of the website (e.g. modifying the html output from my PHP/ASP-like pages) means an expensive build process too which is pretty tedious.  

So what I want is to have QxCompiler sitting as a background process, watching all these applications and the source files they depend on; when a source file changes it rebuilds the app instantly (e.g. 250ms per affected application) and notifies Tomcat to reload the server code if necessary.  At the moment, incremental builds take QxCompiler approx 700ms including the (not insignificant) overhead of starting up node.

Re requirements: I’ve updated the docs to match

I’m working on QxCompiler again this morning and should have an update for you by lunchtime 

Cheers
John

From: Dimitri <[hidden email]>
Reply-To: qooxdoo Development <[hidden email]>
Date: Monday, 15 February 2016 at 18:16
To: <[hidden email]>
Subject: Re: [qooxdoo-devel] QxCompiler - add ES6, faster compilation, and 100% Javascript API to building applications

Hi! Tried again - things are much better now :) However, two issues still remain, one of them being resource management.

Indeed, test app loads its icons correctly in both versions. Unfortunately, this is not the case for my app. What I've found yet:

- the app loads all the qx.* resources correctly (window minimize/maximize/close icons etc.);
- none of the application's own resources could be loaded. The request is made to "build-output/myapp/foo.png", which obviously results in a 404.
- in resource-db.json, all resources that *do* load contain non-empty metadata block, while those that do *not* have an empty literal {}. The same could be observed in qxt for com/zenesis/qx/upload/* resources.

(Please note that our app contains not only images, but audio files in different formats, mainly mp3/ogg. We put them into "resource/sounds".)

The app still cannot load. That unmarshalling error seems to have gone, but now I've got a "too much recursion" somewhere around the same location in code. (stacktrace)
I'll try to produce MWEs for both issues later today. Some other stuff to consider:

1. NodeJS <= 0.12 is not supported, while 4.x and 5.x are fine (due to object literal extension used at lib/qxcompiler/ClassFile.js:329). Some distros still ship NodeJS 0.x with their stable versions. I think this should be mentioned in the docs. OTOH, why not babelize QxCompiler itself? :)
2. What practice would you recommend for using QxCompiler on a build server? For each build, a clean checkout is made. Doing npm install in the project directory is very expensive (~15MB deps). Assuming QxCompiler resides in its own directory, what would you recommend? symlink <project>/node_modules -> <QxCompiler>/node_modules? install deps with npm -g and use <a href="https://linklocal? ">linklocal ? Ideally, I'd like some practice that wouldn't change when we finally have a NPM distro of QxCompiler.
3. It's worth mentioning in the docs that ImageMagick is required (due to "identify" binary being used in qxcompiler/ResourceManager.js).

Cheers,
Dimitri

------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________ qooxdoo-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
qooxdoo-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
1234
Loading...