Re: [pgadmin-hackers][discussion] Javascript package management (Yarn?) - Mailing list pgadmin-hackers

Hi

On Fri, Mar 3, 2017 at 8:36 PM, Atira Odhner <aodhner@pivotal.io> wrote:
>> - Whilst it cuts down on the bloat caused by unnecessary files in the
>> source tree, it does nothing to prevent the problem in the packages we
>> distribute to users (installers, app bundles, RPMs etc) which are the
>> primary concern in this are (I can live with bloat in the source tree,
>> but minimising bloat in the final packages is high priority).
>
>
>  That’s a good point. I hadn’t really thought about that. Requirejs pulls in
> dependencies at runtime instead of using a transpile step so all “potential”
> dependencies have to be available in the end packaging.
>
> Most of the frontend-heavy applications I’ve worked on in the past couple
> years have moved to using tools like browserify, babelify, etc instead of
> requirejs so that dependencies get pulled in during a transpile step and the
> whole thing gets (optionally) minified. The end result is that the deployed
> javascript bundle ends up being much smaller.
>
> I’d like to do that with pgadmin’s javascript. To do that however, we would
> need the javascript files to be something that could be understood by a
> transpiler, meaning we would need to pull the flask templating out of them.

I'd like to find a way to do that, however it may be tricky in some
cases. Translations are the most common issue (and could be fixed),
but there are some other instances as well.

> I’d really like to move in that direction and start creating a separation
> between the backend and the frontend. We can start to do it piece by piece.
> That’s part of what I had in mind with the translate.js/translation.js
> change.

Hmm, that make me realise what was making me nervous about the
translate.js change that I couldn't put my finger on. The code is
intentionally modular right now - you can literally drop in a package
containing the code to handle a new object type (for example), and
it'll just work. Each node is intended to be self-contained.

If we centralise translations, or re-arrange frontend vs. backend code
(in certain ways), we lose that. That modularity was one of the
original requirements of the project, and isn't going to change, so we
need to keep that in mind.

>> - It's using the NPM repository, and as far as I can see doesn't offer
>> a way to add packages that aren't in NPM (short of publishing them to
>> there ourselves). This means no wcDocker, no aciTree and no
>> Filemanager (though - on reflection we may have actually forked that
>> one). We'd either have to have users manually add those, or continue
>> to keep them in-tree.
>
>
> wcDocker is available in npm:
> https://www.npmjs.com/package/webcabin-docker

Ah, OK - I was searching for wcDocker and drawing a blank.

> npm/package.json can be pointed at github repos or even arbitrary web
> resources. I don't know if the same is true of yarn.
>
> Maybe we should try out both yarn and npm and see how that affects our
> package bloat.

Bear in mind that bloat isn't just a size thing, but also quantity. A
thousand 1KB files are arguably worse than an extra 1MB file.

> On Fri, Mar 3, 2017 at 6:34 AM, Dave Page <dpage@pgadmin.org> wrote:
>>
>> Hi
>>
>> On Thu, Mar 2, 2017 at 6:11 PM, Atira Odhner <aodhner@pivotal.io> wrote:
>> > Hi Hackers,
>> >
>> > We've had some discussion on another thread in the past couple weeks
>> > about
>> > pulling in npm to manage our javascript dependencies.
>> >
>> > I wanted to pull this discussion back up to the top level and hopefully
>> > get
>> > some more input.
>> >
>> > We recently moved the external front-end dependencies into a /vendor
>> > directory to give some much needed clarity between what is pgadmin code
>> > and
>> > what is vendor code, but we've continued to see some specific pains
>> > around
>> > the javascript package management.
>> >
>> > We discovered that the SlickGrid dependency was out of date and also had
>> > a
>> > line commented out which prevented an event from firing. That kind of
>> > modification to a vendor library can make it very difficult to
>> > understand
>> > why the code is not behaving as expected.
>> >
>> > When we went add a dependency on jasmine, Dave pointed out that we are
>> > checking in lots of files that are not necessarily used but typically
>> > come
>> > packaged with jasmine. My thinking was that it would be easier to
>> > upgrade
>> > the dependency in the future if it was a simple drop-in and there was no
>> > need to customize and pick apart what was in the directory. That said,
>> > it
>> > does add extra bloat to our codebase when dependencies are checked in
>> > wholesale.
>> >
>> > Still, it brought back to mind that these issues would be resolved if we
>> > were including the libraries using a package management tool.
>> >
>> > libraries.txt appears to be a lo-fi manual package management system. It
>> > has
>> > a record of the licenses that correspond to each library. If there is
>> > concern about using a package manager and picking up libraries with
>> > incompatible open source licenses, we could consider using a tool like
>> > LicenseFinder to run as part of our build and ensure that libraries we
>> > are
>> > depending on have licenses that make sense for the project.
>>
>> Actually that file is there because we (EDB) often have customer's
>> legal departments demand lists of all the Open Source code and
>> projects that we use in any of our products. By listing the bundled
>> packages there we save ourselves (and any other Postgres companies
>> that distribute pgAdmin or a derived product) the pain of having to
>> regularly collate an up to date list.
>>
>> > Dave expressed concerns that this would be adding an extra burden to an
>> > already complex development setup process.
>> >
>> > Looking around at package managers, I wonder if Yarn might better
>> > address
>> > some of the concerns about dependency availability and avoiding adding
>> > extra
>> > steps to a build process. It's provided by Facebook under the BSD
>> > license.
>> > It looks like they had some of the same concerns about wanting to check
>> > in
>> > their dependencies but not wanting all the package bloat that would come
>> > with checking in a node_modules directory. Thoughts?
>>
>> It certainly looks good in a quick look, but I still think there are
>> issues to consider:
>>
>> - Whilst it cuts down on the bloat caused by unnecessary files in the
>> source tree, it does nothing to prevent the problem in the packages we
>> distribute to users (installers, app bundles, RPMs etc) which are the
>> primary concern in this are (I can live with bloat in the source tree,
>> but minimising bloat in the final packages is high priority).
>>
>> - It's using the NPM repository, and as far as I can see doesn't offer
>> a way to add packages that aren't in NPM (short of publishing them to
>> there ourselves). This means no wcDocker, no aciTree and no
>> Filemanager (though - on reflection we may have actually forked that
>> one). We'd either have to have users manually add those, or continue
>> to keep them in-tree.
>>
>> In short, I can see how yarn solves some of these problems for us, but
>> it doesn't solve all of them. The first will bloat packages, and the
>> second will require us to continue with some things in-tree in all
>> likelyhood (though at least two of the possible 3 of those are
>> modified anyway).
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>
>



--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company