worknotes - helloworld with db

Ok,
Looking back at my smaller steps plan blog post:

[1] helloword-cookies (cookieparser?).  DONE
[2] helloworld-db (mongdbhq?).  NOT
[3] helloworld-routes (express) ALSO DONE
[4] helloworld-session (?). NOT
[5] helloworld-authentication (passport, passport-local) NOT

It seems that the primary change from the original plan was the earlier introduction of express to allow writing the hello world following dglittle's code-style: ie make the index.htmk be just js functions that use apis to figure out the state and create appropriate displays. The server side APIs involve routing which brings the need for express.

So the next step would be to add the db layer.
In this simple next step we would
- on register
+ use the db to enforce username uniqueness
+ store the userinfo in the db
+ set the username in the cookie
+ return the complete userinfo
- on logout
+ reset the username cookie
- add a login screen that allows the user to switch to a prior user context.
+ checks for username existance and if found sets the username to the cookie
- add a change user info screen that is prefilled with the userinfo and allows one to change
the user to change first or last

• Here is the github repo : https://github.com/ogt/helloworld-node-db
• Here is the heroku app : http://helloworld-node-db.herokuapp.com (20+ sec delay at first fetch)
• Here is the blog post about this : http://otdump.blogspot.com/2013/03/worknotes-helloworld-with-db.html

I will use mongolab as the db.
Ok it seems that
> heroku addons:add mongolab
(which is equivalent to heroku addons:add mongolab:starter) ie free plan for mongolab)
is doing all the right things. It registers the add-on - and it seems to be somehow associated with the specific app. I am actually not sure what exactly that means.
I hope it means that I can get a free starter plan with every one app (as opposed to just one for my account). I hope it doesn't mean that I can't access the mongodb from two separate apps if I wanted by sharing the URI.  Even though there was a UI to add the mongo lab addon, the ability to do all that just from the command line  (with the command above) feels so ... good and powerful.
Befor I added the db code, I wanted to familiarize myself a bit with the db - I thought of using the mongo shell for that.
First attempt failed - it doesn't take heroku uris as parameter.
> heroku config|grep mongo 

MONGOLAB_URI: mongodb://heroku_app1234567:some_secret_pass@ds0123456.mongolab.com:45432/heroku_app1234567

What does the man page say?  no man page.. Googl-ing instead I find some docs, re-trying it, 

after 4 efforts and going back and forth googling for docs I am really annoyed that there is no man page.

How could it be that there is no man page? I just brew installed it. Maybe the manpath isn't right..

No, there is no mongo* in /usr/local/share/man/*/.

... 1 hr later... I have man pages.. it took quite some doing:

> brew install sphinx  #that probably wasn't needed

> pip install sphinx  # building the docs requires sphinx-build

> cd /tmp

> hub clone mongodb/docs

> cd docs

> make man # fails with some docutils error  #active bug

> pip uninstall docutitls # remove current docutils

> pip install docutils==0.9.1  # last 

> make man

> cd build/master/man/man1

> man mongo.. #yeahhh

> cp * /usr/local/share/man/man1

> pip install --upgrade docutils # bring back the current version

> rm -rf /tmp/docs #cleanup

Luckily right after that I managed to get mongo to connect:
> mongo -u heroku_app1234567 -p some_secret_pass ds0123456.mongolab.com:45432/heroku_app1234567
..
and I got in

Work notes - helloworld with cookies

02/23/2013..
The plan here is to use cookies to save something useful, something that resembles the signup information.
How about the user provided inputs
 - username
 - first name
 - last name
Ok, there is even a semi ready form for that somewhere,.. yes passport-local. Copied that, changed it a bit, appeneded at the end of my hello world.
Oops, problem. Can I submit a form?. It seems I get an error. Should I attempt to grab the user info myself - e.g. by change the method to get from post and parsing the string and the destination to /.. or should I add express as a minimal routing middleware. Lets try the do it yourself approach first.


Ok - not an error any more and I see the url params. good. Seems I made the right choice. Of course it doesn't refill in the form the parameter in the url.


Lets how I can get the url params from the server side. It seems that the connect-query does just that - it creates the req.query object.

Lets add it. Added it. Hmm,... now where do I put the code for this to do something.... darn, still I need some url receiving place ie a route.

Its not right to just add this as a middleware function. How does gl does it. He is using express.. should I bite the bullet and use express? already? and do I need connect.query then? or express does this for me?

Looking at the express documentation it seems that express adds connect with cookies and qs, qs could be sth like query
http://expressjs.com/guide.html

$ npm ls
hello-world@0.0.1 /private/tmp
└─┬ express@3.0.0beta7
  ├── commander@0.6.1
  ├─┬ connect@2.3.9
  │ ├── bytes@0.1.0
  │ ├── cookie@0.0.4
  │ ├── crc@0.2.0
  │ ├── formidable@1.0.11
  │ └── qs@0.4.2
  ├── cookie@0.0.3
  ├── debug@0.7.0
  ├── fresh@0.1.0
  ├── methods@0.0.1
  ├── mkdirp@0.3.3
  ├── range-parser@0.0.4
  ├─┬ response-send@0.0.1
  │ └── crc@0.2.0
  └─┬ send@0.0.3
    └── mime@1.2.6

> npm ls

└─┬ connect@2.7.3

  ├── buffer-crc32@0.1.1

  ├── bytes@0.2.0

  ├── cookie@0.0.5

  ├── cookie-signature@0.0.1

  ├── debug@0.7.2

  ├── formidable@1.0.11

  ├── fresh@0.1.0

  ├── pause@0.0.1

  ├── qs@0.5.1

  └─┬ send@0.1.0

    ├── mime@1.2.6

    └── range-parser@0.0.4

!.. Interesting! I already have cookies and this qs things. But why cookies. Maybe because static needs it to keep track of information about file dates etc? maybe. No need to know really (yet). Lets find what is this qs thing..

03/02/2013. Ok One week later with barely any time for serious work in between , still with several micro-iterations I am pushing my final version of this iteration

Register works and Logout  work - I also made the whole thing look a bit more like what I had in mind originally.
Decisions made along the way
- Used dglittle's libraries from https://github.com/dglittle/nar-nar
  + u.js
  + rpc.js
  + nodeutil.js
- Used dglittle index.html base code. This includes
  + the client side of rpc.js,
  + the coding style/methodologie
- Added express's routes, and connect's cookie parser - plus all the prior middleware
- Implemented a few basic rpc methods (register,logout,getuser) and used a single file index.html as the client side driver.

• Here is the github repo : https://github.com/ogt/helloworld-node-cookies
• Here is the heroku app : http://obscure-bastion-8852.herokuapp.com/ (20+ sec delay at first fetch)
• Here is the blog post about this : http://otdump.blogspot.com/2013/03/worknotes-helloworld-with-cookies.html

Work notes - loginator - smaller steps

Ok.
It seems that what I thought as the next step, ie, a system that simply allows signup/login is a bit too big of a jump, given the level of my ignorance. So I will create a sequence of helloworlds to move more incrementally.

A system that uses passport-local seems to need:
  - a db to save the information,
  - a session to remember that you are logged in,
  - cookies to allow for the session to be identified, and possibly
  - some basic routing since we have at least two urls /login and / .
Thats a lot.
So here are the steps that I would follow (found by mentally unraveling the dependencies of each of of the steps above)

- helloword-cookies (cookieparser?).
I would first add to my hello world the cookie functionality possibly keeping track something like the value that I entered in a form asking for a userid and my name (First, Last). After my first entry, the page should be able to display who I am and still allow me to  add a new identity
by re-filling the form.

- helloworld-db (mongdbhq?).
Then I would add a db to allow saving that information  (first, last) in the db keyed by my userid, save only the userid as a cookie. In the case the form would display who I am, allow me to switch identities to someone else by picking a new userid that exists already in the db, or add a new identity all together

- helloworld-routes (express)
I would break the single page site into multiple pages /index.html being the form that displays who
is currently logged in and gives links to logout, login-as-someone-else, register

- helloworld-session (?).
Then probably the next thing would be to see how I could use db backed sessions and session identifiers as the cookie in that case. In that case when I create the session I can store in there the name that I fetched from the db, and rely on that for subsequent fetches.

- helloworld-authentication (passport, passport-local)

Work notes - using middlewares

Ok,
It seems that my simplistic web server is the perfect vehicle to help better understand middlewares in node.js. I understand all its pieces, and its obvious that every part is minimalistic/not sufficient for a production site,
 - logging,
 - mapping of url space to local filesystem directories,
 - handling of default files,
 - handling of content types,
 - handling of error responses,
 - handling of HTTP headers.

Each one of these represent a few lines of code in the 50-lin web.js files.
So what I think is an interesting exercise is to incrementally replace each one of those things with a connect middleware and see what changes need to happen in the process and what is left over in the end. What I am expecting to get from this exercise is
 - a more robust, production-level hello world
 - a better understanding of connect, middlewares
 - a system that I can add in the next step basic authentication, session management to build my simple utility app that I set as my goal.

Before proceeding a decision,  should I continue working on the existing repository?
Hm, the blog points to the files, but I can move the files as "examples". But the heroku setup (package.json, etc ) is what it is because we have no dependencies. Hm., it seems that in node.js/heroku the natural unit is the repository. Then I guess I will have many repositories, possibly one per blog post.
I should probably go back to the last post and leave a link to the repo + a link to the heroku app.
Done - Added links both in the repo's readme and in the blog

Creating a new repo - I guess the folder's name is important - it will be the repo's name using our hub create method.... copying all files in. and back into connect documentation.
Actually lets first try and make sure that everything is working fine before any changes - not very confident yet that I have mastered the create repo/github/heroku thing....
Ok following the previous steps

> # created a copy of the folder files with new folder name... 
> # from inside that folder

> npm install
> git init .
> git add . ; git commit -m 'init'
> hub create
> hub push -u origin master
> heroku login
> #test foreman start..  open ...
> heroku create
> git push heroku master
> #test open ...

I have now :

Here is the github repo : https://github.com/ogt/helloworld-js-heroku-depends
Here is the heroku app : http://guarded-stream-2218.herokuapp.com/  (20+ sec delay at first fetch)
Here is the blog post about this : http://otdump.blogspot.com/2013/02/work-notes-using-middlewares.html

First step see what it takes to use the connect createServer without any of the middlewares.
Hopefully it will be exactly the same as using the http's createServer method.
Making the following changes in two files (not very sure about what version I should use for connect..)

then running
> npm install connect
> #test foreman start.. open
> git add . ; git commit -m 'replaced http with connect'
> git push origin master
> git push heroku master
> #test open ..

and everything is still perfect besides the massive upload of dependencies to our github repo.
Next step, lets replace the loggin command in web.js with the logging middleware.


I will actually use  logger('dev') formatting option to get the color highlighting in my terminal
Next step, lets try to remove the error handling code.

Hm.. the error code seems intertwined with the file service code... may be I should do that first.
Apparently I will need to move my index.html file under public and set public as my static file serving folder.  Ok, I did that and added

+ connect.static(__dirname + '/public')

as well. Things work, and it seems that they serve off the public folder. But it worked so smoothly.. I am not sure. I start removing lines from the web.js and keep on testing it.
 - removed the contenttype code , still it works
 - removed the error handling code, still it works
 - removed the index.html defaulting, as well as the mapping to file path code... still it works
 - removed the complete function we added to createServer... still it works

At this point I started having doubts... could I be testing the wrong site - it seems to be working no matter what I do. I renamed index.html in the folder. I get an error. Maybe I should add an image to my hello world to make sure that the content type is set correctly.. did that still it works.
With every fetch I keep on getting the error code for the favicon - I haven't done anything about it... so I know the error handling works. Also I see a 304 ... what is that... it seems that the static middleware is "smart" and returns a "not changed" status code instead of resending the file every time..


Could that be it? I refresh the browser with Shft-Command-R and I get 200s!! Perfect. Everything works perfectly.

Maybe I should just fix the favicon as well. I think I saw a middleware for that.

I add it. The error disappears in the log. But no other difference. I should be seeing the "default connect icon". Don't see it. Maybe I should set a custom one. How to do that... found an online convertor.. convert my portrait to an icon and save it as favicon.ico. No diference. Maybe its because its in my localhost? I push it in heroku. Refresh and now I see it. I have been looking at the wrong place....all this time - the favicon is at the browser tab - not left of the url input box.. What was I thinking...

I push everything to github (commit).

I think I am almost done. Maybe I should add some decent css.
Need to use something modern.. Was it blueprint? compass? bootstrap? I browse a bit before I conclude   to bootstrap..  download the files copied them under public and with a couple small changes in the index.html file


we are done at last!!  (commit)

Work notes - a minimal node.js heroku app

Here are the final instructions: (prerequisites brew instal node npm hub, and minimal web server )
(trying to see if embedded gists are better ... they look better but its too much work for each little piece of code....)  

Here is the github repo : https://github.com/ogt/helloworld-js-heroku/
Here is the heroku app : http://calm-taiga-3864.herokuapp.com/  (20+ sec delay at first fetch)
Here is the blog post about this : http://otdump.blogspot.com/2013/02/minimal-nodejs-heroku-app.html

Here is how I found them:

In my machine here is what it take to run the most minimal node.js app:

> mkdir ~/Dropbox/Home/code/node/helloworld
> cd !$
> cat > index.html
<html>
<body>
hello world!!!
</body>
</html>
> node ../simplewebserver/web.js &
Server running at http://127.0.0.1:8081/
> open http://127.0.0.1:8081/

Pretty simple (given that I have already brew installed node/npm).
Now you may ask what the heck is simplewebserver/web.js? Its the simplest no-dependency requiring file serving node.js code I have found from some prior node.js play-attempts of mine.
Here is the code:

> cat ../node/simplewebserver/web.js 
var http = require('http');
var fs = require('fs');
var path = require('path');

http.createServer(function (request, response) {

    console.log(request.url)
     
    var filePath = '.' + request.url;
    if (filePath.match('/[^.]*$')) filePath += '/'
    if (filePath.match('/$')) filePath += 'index.html'
         
    var extname = path.extname(filePath);
    var contentType = 'text/html';
    switch (extname) {
        case '.js':
            contentType = 'text/javascript';
            break;
        case '.css':
            contentType = 'text/css';
            break;
        case '.jpeg':
            contentType = 'image/jpeg';
            break;
    }
     
    path.exists(filePath, function(exists) {
     
        if (exists) {
            fs.readFile(filePath, function(error, content) {
                if (error) {
                    response.writeHead(500);
                    response.end();
                }
                else {
                    response.writeHead(200, { 'Content-Type': contentType });
                    response.end(content, 'utf-8');
                }
            });
        }
        else {
            response.writeHead(404);
            response.end();
        }
    });
     
}).listen(8000);

console.log('Server running at http://127.0.0.1:8000/');

Ok, What does it take to get that minimal hello world app running on heroku?

Following the instructions at github
> mkdir ~/Dropbox/Home/code/node/heroku-helloworld
> cd !$
> cp ../helloworld/index.html .
> cp ../simplewebserver/web.js .

> heroku login
> cat > package.json
{
  "name": "node-example",
  "version": "0.0.1",
  "dependencies": {
  },
  "engines": {
    "node": "0.8.x",
    "npm": "1.1.x"
  }
}
> npm install
npm WARN package.json node-example@0.0.1 No README.md file found!
> cat README.md
Simplest possible hello world heroku node.js app
> npm install
> ls
/                    README.md             package.json
../                  index.html            web.js
> cat > Procfile
web: node web.js
> foreman start
12:19:11 web.1     | started with pid 35582
12:19:11 web.1     | Server running at http://127.0.0.1:8000/

> open http://127.0.0.1:8000/

...
12:19:30 web.1     | /index.html
12:19:30 web.1     | path.exists is now called `fs.exists`.
12:19:30 web.1     | /favicon.ico
^C

# edit web.js and replace path.exists with fs.exists
#refresh browser

...
12:20:42 web.1     | /index.html
12:20:42 web.1     | /favicon.ico

ok things work normal locally
lets add to git 


> cat > .gitignore
.*
!/.gitignore
> git init
> git add .
> git commit -m "init"
> heroku create
Creating calm-taiga-3864... done, stack is cedar
http://calm-taiga-3864.herokuapp.com/ | git@heroku.com:calm-taiga-3864.git
Git remote heroku added
> git push heroku master
..... (lots of stuff)
http://calm-taiga-3864.herokuapp.com deployed to Heroku

To git@heroku.com:calm-taiga-3864.git
 * [new branch]      master -> master
> heroku ps:scale web=1
Scaling web processes... done, now running 1
> heroku open
Opening calm-taiga-3864... done

And here I get a crash
Logs show just that the app crashed.

> heroku logs
2013-02-17T20:29:29+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path=/ host=calm-taiga-3864.herokuapp.com fwd="50.0.42.114" dyno= queue= wait= connect= service= status=503 bytes=

I repeat while running heroku logs tail

> heroku logs --tail
> heroku ps:scale web=1
> heroku open

and I find the offending entry :

..
2013-02-17T20:28:54+00:00 app[web.1]: Server running at http://127.0.0.1:8000/
2013-02-17T20:28:54+00:00 heroku[web.1]: Error R11 (Bad bind) -> Process bound to port 8000, should be 10472 (see environment variable PORT)
..

So it seems that my simple web server with the hardcoded port is not what heroku likes. The heroku environment wants to have control of the ports... Makes sense given that open heroku sends to port 80 so someone will need to redirect to a pre-known by heroku port. I grab the code from the suggested hello world ap and add it to my web.js

> grep -in PORT web.js
4 :var port = process.env.PORT || 8081; 
47:}).listen(port);
49:console.log('Server running at http://127.0.0.1:'+port+'/');
> 

I git update, git push to heroku and everything works perfectly this time !
Last step is to push my git a githib repo so as I can point to the various files from this blog post.

# fixup the enclosing dir - so as it matches the repo name (to make hub happier)
> mv ../heroku-helloword ../helloworld-js-heroku
> hub create -d 'hello world app in node.js for heroku'
> git push -u origin master
Counting objects: 11, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (8/8), done.
Writing objects: 100% (11/11), 1.41 KiB, done.
Total 11 (delta 2), reused 0 (delta 0)
To git@github.com:ogt/helloworld-js-heroku.git
 * [new branch]      master -> master
Branch master set up to track remote branch master from origin.
> 

Done!!