Open sourced nodejs web apps - missing the best practices

While it has been obvious that people open source modules, in every single language, the idea that when you write a deployable web (or mobile or cli) app you should be
 - open sourcing it
 - link the app with the open source code of it
is much less discussed.

Here are a few examples
1 npmjs.org -> https://github.com/isaacs/npm-www
2 travis-ci.org -> https://github.com/travis-ci/travis-ci

3 liftie.info -> https://github.com/code42day/liftie

4 modulecounts.com -> https://github.com/edebill/modulecounts

The practice is so uncommon that there aren't "best practice" guides for this, "this" being not the writing of the web app itself but the open-source process:

- What should I include in the public repo and what should I not include?

- Should I add an overly visible link from the deployed site to the code (fork me at github) or should I leave it for people to search and find the source.

...This is a very important point - equivalent to package your open source with the binary in the old GNU inspired days..

I often find a site that has a feature/functionality I want - but I want to make it a bit different. The site seems to be written by someone with open source ideology (e.g. 1,2,4 above). So how do I find the source?

I google "github <domain>" and if that doesn't work I typically give up.

Note that googling link:liftie.info finds no results even though the github repo is pointing to the site as its web url. Either google is increasingly screwing up its advanced options or github doesn't expose this well for search engines...
And I don't think that there is a "registry" for open source apps of sorts. You cannot find npmjs.org's code in npm (I am guessing that...haven't really looked)

Anyway but that is just one type of "best practices". What do you do about the data? For example in the modulecounts  site above the author has opted to put in the DB the regexpressions/urls that extract the counts from the registries... Thats a good part of the logic. Luckily he has checked in the db migration folders as well !!! which gives access to everything one would need. But there is more.

Lets say I want to take modulecounts and just write it in nodejs for the fun of it. Or even more I want to add some of the charts that Mikeal pointed out in his post, that show growth or proportion growth.. To do that I would need the actual historical data that the service has collected.

It would be nice to have that. Modulecounts again did the right thing and gives access to the csv in their  web site! But of course the app shouldn't/couldn't share private user data...

Another area of "best practices" is dependencies and testing 

It would be nice for the web app to have a test suite and travis-ci integration.

We would be able to create dependency graphs not just for modules but for actual apps!

It would give tremendous test coverage for all the module writers to confirm the validity of their changes . 

In general as I have said in an earlier post we need best practices that allow us to produce truly open apps. And we need people that are influencers to start following the practices and publish their apps as well (and not fear that by doing so they risk losing their userbase to someone else ) even if the apps are "for profit" apps.

More on nodejs dependencies and nodejs uniquenesses

continuing on on comments made here

http://www.futurealoof.com/posts/momentum.html
This article is the analysis of why nodejs is (and will be ) a more successful platform than its contemporaries (python ruby etc). 

...In the earlier days of node you could write anything. The advantage of nothing being compatible is that there is a much greater opportunity to attract people interested in writing virtually anything. That means a greater potential for early investment, which creates early momentum, which you can only lose over time so maximizing that early growth is the most important thing....


(my comment: one thing to note here is that Java had a similar potential early on. The virtual machine created a very isolated world where no existing (C/C++) lib could be brought in. However, that was seen then as a disadvantage and the  "parents" of that platform attempted to alleviate it by expanding a massive "standard" set of libraries that emulated the previous world and choked any potential for early on community investment.)


Other interesting article from this guy's blog:

http://www.futurealoof.com/posts/nodemodules-in-git.html
Summary of that post:

If you are building an app or a web service or anything that is "deployed" then you should include node_modules in the repo but not binary targets 

If you are building a package/module (ie sth that is used by other packages/modules or apps ) then do not include the node_modules in the See for example 

deploys:

https://github.com/isaacs/npm-www

or packages

https://github.com/isaacs/node-tap

Bold statements

Reading npm's design philosophy , I found this statement to be rather bold and rather representative of the module mentality seen pervasive in the nodejs community:


- No lib folder : If you would put it in lib/, then it belongs in a separate module.

Here are a few other stmts as well

Design Philosophy

- No frameworks : Everything is done using small, simple, standalone modules that work with vanilla Node.js http servers.

...

Small Modules : No single JavaScript file should be more than about 200 lines.

...

Unceremonious MVC : No big MVC class heirarchy. Just have the route handler get some data, then hand it off to a template. Simpler is better.

Chaos vs Order (Law)

As I read the various denominations/alignments of dungeons and dragons I think that I am more of a "chaotic good" creature....

Work notes - HS/GH evolving the flow

I am applying the moto "what other services/modules/libraries that are even remotely reusable/generic if they were to exist would make the job of implementing my project trivial"... I reached the following realization.
substack:
Which hypothetical reusable modules would make the task at hand trivial?

When you do that the first thing that happens is that you find other modules that are doing something similar to what you need but not exactly. So then you have the following options:

 1 Ignore the existing service and do something simpler yourself that does just what you need 

 2 Talk with the owner to see if the needed feature/expansion of functionality is sth that makes sense for that module. If yes:

    a ask the module owner if they would do it. (Even if the person is willing to do it, you are often being blocked by the missing functionality which means you probably need it asap which makes that option... rarely an option

    b fork the repo and try to do it yourself

 - If no

    c you may still fork the repo and create a variant of the module that provides that new functionality and publish it 

2c and 1 are similar. However, only 1a is truly leveraging others. I think it is very important to figure out how to better enable 1a (as opposed to just allow someone to outsource the todo list of their own repo).

So here is how we could do that.

The existing model of HS/GH  is as follows.

- You register one repo with HS/GH

- You can request for any of the issues in your own repo to be od sourced (by providing some additional info around the issue (budget, tat, primary skill, secondary skills) (you can provide these info either as back-matter yaml in the issue itself or directly in an external system. By doing so the following things happen

   - The issue owner is charged the amount

   - An od job post that links to the issue gets created based on the information above

   - A pool of candidates get automatically invited

   - Applicants that fail to meet requirements are auto-rejected

   - The first applicant that meets the requirements is auto-hired, and their gh username is linked to the job, and the job becomes filled

   - On pull request from the the linked user the system confirms that acceptance criteria are met. If not user is notified that the issue is not ready yet.

   - When the owner accepts the pull request payment is being made and job closes

   - Meanwhile a issue progress icon is maintained that can displayed in the issue's markdown both to help the issue owner as well as to attract more/future candidates.

   - If time passes past the tat, notification is given to issue owner to extent the deadline or the job gets cancelled and the money is returned to the issue owner.

The owner is freed from the hiring/paying process.
The owner still has to manage the person, answering questions, reviewing the pull request etc. 

He can avoid some of that by putting more time in the issue details. 

He can avoid some of that by following strong unit testing, coverage, lint enforced practices etc.

Still he has to put a non-insignificant amount of time, limiting essentially the extent to which a person can truly scale themselves.

To enable muti-level scaling we need to better faciliate the 1 case above.

So here is an idea of how to go about doing that:

HS/GH also supports the ability to "pledge" towards the implementation of issues/features on a repo that someone is not an owner of (like sourcebounty .com)

You can pledge an amount of $s towards the implementation of an issue. 

To figure out how much to pledge you really need to ask the repo owner - how big of a job it would be for you to do the job (assuming guidance by the owner - which in most cases they would be willing to provide). The amount of hours x rate is an indication of the pledge amount.

A simple way to think about that (similar to what we did above with the issue yaml back-matter) is to allow a pledge to be given in the comment of the issue as comment back-matter yaml. Essentially you link a particular comment on an issue on someone else repo with a pledge (amount+ TAT). That does the following:

 - checks if there is already an od job for this issue. 

 - If yes it augments the od job with the extra pledge/amount/tat.

 - if not creates the od job as private and invites the repo-owner as sole applicant.

 - the owner accepts the invite (no obligation) and subsequently can do 2 things

   - creates (one or more related issue and gets it done by someone else following the model above

     (note that he now has the money to do that)

   - does the job themselves and makes the money

     - closes the issue

     - on issue close the payment is being made

 - the pledge should have probably room to allow the repo owner to make some money themselves (even if they outsource the job) to increase the motivation and get the job done on time.

This means that module/repo-owner would make money by resolving issues. The more users a module has the more issues it would create. The more modules someone maintains the more issues.

The model allows a developer to make more and more money by developing the stuff that want to be writing.

Work notes - weekend planning

Next things in my quest for perfection for the small first project.
1. Still not satisfied about the man pages. mantastic didn't do a great job. Sections are not highlighted etc.
2. Cover requires a single file to run - most programs should be able to take a list of files. Should make a change. HS it?
3. Cover is not available as a library - make it a library. HS it?
4. Add unit tests for jshint and cover and include them in the test program. Figure out how to create a test that "does not decrease code coverage".
5. Still not satisfied with the Makefile. Should probably have build, cover data created as part of the build process and test depend on it. Make test should be smart and do everything by understanding file dependencies
6. Too much clutter at the top level. Some files should move down folders. maybe some folders should merge.
7. Issue HS jobs for the new brush types.
8. Change my github ssh  defaults so as when I push, I push as ogt (as opposed to tsatalos). Do that without messing amzn/heroku.

Browsing hacker news

https://github.com/f/omelette ->
The interesting thing here is the use of animation gif in the github read me….

http://blogging.alastair.is/how-i-served-100k-users-without-crashing-and-only-spent-0-32/
The interesting thing here is the realization that the gh-pages feature of github may not be worthwhile the effort (and the various annoyances) . An S3 wrapper to push your static site to S3 on commit may be a better option.

I just submitted my first hacker news comment.. https://news.ycombinator.com/item?id=5613652
and subscribed to http://hnnotify.com/ and imitated the animation gif on boxchareditor.

Happy.