I wanted to convert a very useful unix utility (cloc) into a web service.
I thought, that someone may have already created a little glue that can do that.
Especially given nodejs and the recent love that streams are getting there (https://github.com/substack/stream-handbook) together with the re-focus back into the unix small utility programs fit together with stdio/outs.. etc someone must have done it.
Anyway, so far I can't find anyone.
The only thing I found is someone 5yr old blog and broken references (guthub and gists weren't that popular then .. so no trace of whatever this guy did...)
http://swik.net/Unix/BASH+Cures+Cancer+Blog/Exposing+command+line+programs+as+web+services/b3x8u
This guy seems to have the right idea:
I thought, that someone may have already created a little glue that can do that.
Especially given nodejs and the recent love that streams are getting there (https://github.com/substack/stream-handbook) together with the re-focus back into the unix small utility programs fit together with stdio/outs.. etc someone must have done it.
Anyway, so far I can't find anyone.
The only thing I found is someone 5yr old blog and broken references (guthub and gists weren't that popular then .. so no trace of whatever this guy did...)
http://swik.net/Unix/BASH+Cures+Cancer+Blog/Exposing+command+line+programs+as+web+services/b3x8u
This guy seems to have the right idea:
$ ./to_web.py -p8008 sort & Thu Mar 27 13:45:54 2008 sort server started - 8008 $ ./to_web.py -p8009 gzip & Thu Mar 27 13:46:29 2008 gzip server started - 8009
Use the services:
$ for i in {1..10}; do echo ${RANDOM:0:2}; done | \
> curl –data-binary @- “http://swat:8008/sort+-nr” | \
> curl –data-binary @- “http://swat:8009/gzip” | \
> gunzip
97
37
23
23
21
18
11
11
10
10
-
Note that the to_web.py, seems to be doing just a popen of the command passing the stdin/stdout of the server to the command. Even the URL query string is passed intact as ... no need to think much about mapping cli parameters to url params.
The whole thing looks like substack's way of doing things.
(without any extra that substack may have already added I would use this
http://nodejs.org/docs/v0.4.6/api/child_processes.html as starting point to write the ~10 line to_web.py
... possibly by using .pipe() instead of tie-ing outs and ins manually https://github.com/substack/stream-handbook)
... possibly by using .pipe() instead of tie-ing outs and ins manually https://github.com/substack/stream-handbook)
Any command that is sideffect free and uses just cpu/mem resources, can be made available to third parties... possibly with a bit lower than normal ulimit to prevent abuse... just like travis-ci....
I wonder how many of these commands are part of what heroku exposes... if it does it should be easy to create a free web-servuce version of all unix utilities. All that would be needed would some consistent naming for the domains ... hm and some client side pipe operation (would we need any syntactic sugar??) across the web services.. If I recall correctly streams and related libs can be browserified just fine....
On the other hand Ideally I should be able to hook webservices directly as opposed to be using the browser as the intermediate point.
If I want for example to take a gziped csv file unzipped it sort it and re-zipped.. assuming that I have
gzip.herokuapp.com, gunzip.herokuapp.com sort.herokuapp.com how would I do keeping the stream of bytes going from one webservice to the next....???
One ugly way would be to be calling gunzip with a special option
"send the response to "sort.herokuapp.com" instead of me
But that wouldn't work because then sort would send it to me instead of gunzip.herokuapp.com
I guess we would have to be calling gunzip with a special option
"send the response to "sort.herokuapp.com" instead of me but when doing so with a special option to
send it to gunzip.herokuapp.com" instead of me...
So then the trick is to create a library that does the "syntactic sugar" on top of the ugly thing above so as all I have to say is sth very simlar to
"gunzip.herokuapp.com PIPE sort.herokupapp.com PIPE gunzip.herokuapp.com
On the other hand Ideally I should be able to hook webservices directly as opposed to be using the browser as the intermediate point.
If I want for example to take a gziped csv file unzipped it sort it and re-zipped.. assuming that I have
gzip.herokuapp.com, gunzip.herokuapp.com sort.herokuapp.com how would I do keeping the stream of bytes going from one webservice to the next....???
One ugly way would be to be calling gunzip with a special option
"send the response to "sort.herokuapp.com" instead of me
But that wouldn't work because then sort would send it to me instead of gunzip.herokuapp.com
I guess we would have to be calling gunzip with a special option
"send the response to "sort.herokuapp.com" instead of me but when doing so with a special option to
send it to gunzip.herokuapp.com" instead of me...
So then the trick is to create a library that does the "syntactic sugar" on top of the ugly thing above so as all I have to say is sth very simlar to
"gunzip.herokuapp.com PIPE sort.herokupapp.com PIPE gunzip.herokuapp.com
No comments:
Post a Comment