OS.beginSheetModalForWindow is declared void in Java and jlong in the entry #3546

OS.java declares:

public static native void beginSheetModalForWindow(long id, long sel, long window, long handler);

and the entry in os_custom.c declares and returns a jlong:

JNIEXPORT jlong JNICALL OS_NATIVE(beginSheetModalForWindow)
	(JNIEnv *env, jclass that, jlong arg0, jlong arg1, jlong arg2, FunctionPointer arg3)
{
	jlong rc = 0;

	OS_NATIVE_ENTER(env, that, beginSheetModalForWindow_FUNC);

	rc = (jlong)((jlong (*)(jlong, jlong, jlong, void (^)(jlong)))objc_msgSend)(arg0, arg1, arg2, functionToBlock(arg3));

	OS_NATIVE_EXIT(env, that, beginSheetModalForWindow_FUNC);
	return rc;
}

The VM binds these by name and the mangled symbol carries no return type, so nothing on either side checks that they agree: javac sees only the Java declaration, the C compiler only the definition.

I have not observed any misbehaviour from this and would not expect to. On the ABIs SWT targets the callee writes a return register that a void caller ignores, and rc is unobservable from Java in any case, since DirectoryDialog, FileDialog and MessageBox all call the wrapper as a statement. So this is a JNI signature mismatch rather than a wrong value.

It seems worth mentioning because it looks unintentional and it is isolated: comparing every native declaration in the tree against the return type its own entry declares, this is the only pair that disagrees, out of 3110 comparable pairs at one commit I checked and 2957 at another.

The fix is presumably to declare the entry void and drop rc, unless the value is meant to reach Java, in which case the Java declaration is the side to change.

Checked against master at 3e3b24b4. Found as part of a research project.