There seems to be a bug in Flash 9 Player for Macintosh. The fscommand("exec", "myapp.app") is not working. If you save your movie as a Flash 8 movie and publish it as a Flash 8 projector it works again.

You can download a demo here.

I created a windows application "hello.exe" and a mac application "hello.app". These are little applications that will just display a message box. Both files reside in the "fscommand" folder and should be opened by the fscommand. I also create 2 flash files: exec_demo_fl9.fla to be opened in Flash CS3 and exec_demo_fl8.fla for Flash 8.

The .fla movies have a button with this code:

 open_btn.onRelease = function() {
    if (getVersion().substr(0, 3) == "WIN"){
        fscommand("exec", "hello.exe");
        debug_txt.text = "Trying to open fscommand/hello.exe";
    } else {
        fscommand("exec", "hello.app");
        debug_txt.text = "Trying to open fscommand/hello.app";
    }
}

The button wil issue an fscommand that opens "hello.exe" on Windows platforms and "hello.app" on Mac platforms. 

Now if you publish exec_demo_fl9.fla as a Flash 9 projector, you will see that the Windows version will open "hello.exe", but the Mac version won't open "hello.app". It doesn't matter whether you published the projectors on Mac or Windows. I tested both, it doesn't make any difference.