Thursday, September 30, 2010

Node.js means having to put your toys back in the closet after you're done :(

When I was first exposed to Ruby and how easy it was for functions to accept blocks and how ubiquitous such functions were, I was delighted that for example {|file| foo(file); bar; baz; qux;}
would let me do a bunch of operations on a file and not have to worry about closing the file. This was similar to how cool it was to move from C++ to C# the first time and stop worrying about memory. This was how things should be.
And then recently this post on Stack Overflow led me to this code sample in Node.js. The code is opening a file and appending the word world.

I have to remember to close the file!! In a language that allows functions to be passed as parameters, I have to remember to close the file. How much does that suck? My colleague, Rakesh Pai, a node.js fan, suggested that maybe this is because this is a low level api, and high level apis will take care of this problem.

But on thinking about it, this may not happen. I mean I could imagine a fs.append function that takes a file and some text, appends it and then calls the callback, but not something that gives you access to a file object to play with.
Or maybe could take 2 functions. One meant to be executed after opening the file, and a second to be executed after closing the file. But this will probably cause a lot of scoping grief. So maybe the first function will have to return an array of variables that then become available to the second one?

So it's possible this never happens. If this is the case, is it possible that the file will not be closed for a long time? I mean if your code is something like

Then, wouldn't your workers hang on to the file handle for an awfully long time? Even though it isn't being used? Since the file is trapped in that closure there, there's no way out unless my runtime has analyzed my code and seen that I will never use file from this point on. Not even via an eval. I can't see the GC or whatever figuring this out. How about if say as a good programmer, I do this.

Now can it figure it out, since I haven't passed file to my doJob function. Does that mean that in this scenario I'm safe? I assume it will since file isn't trapped in a closure here. But I don't know how the v8 GC works. But in the previous scenario, it feels to me like it will not be able to detect my intention, so we'll have to be a bit more careful programming with node.js

Edit: From tuxychandru's comment, I realised that I may not have been very clear. I wasn't just whining about some api requiring people to do work. I understand that there will always be low level api's that require more care. I was saying that in some ways node is being positioned as an answer to the programming ills for certain kinds of problems. I was saying that it looks like the paradigm has resulted in a new solved problem returning. That as part of everyday programming, cleaning up is required.


  1. Maybe I'm missing something, but have you looked at `fs.WriteStream`?

  2. don't you have to do a writeStream.end()? Besides I can imagine a highlevel "append" function. Just not one that lets you do the appending inside the callback. (well there is that two callbacks approach I suggested)

  3. The reason Node API can't do it in spite of JS being able to pass functions around is the fact that IO is asynchronous.

    fs.write() returns before the contents are actually written. Ruby's standard APIs are all blocking and thus it can safely assume that the file can be closed when the block returns.

    In your last example, the argument list in function definition in JS is just a convenience. The arguments are passed irrespective of whether the function has said it accepts them. Such arguments can still be used within the function using the 'arguments' variable.

  4. I understand the reason. That's why, in fact, I'm assuming the problem is insurmountable and lamenting that :(.
    I'm fine with some api's requiring people to take care of shit. But i'm saying this might always be required with node and can lead to errors.