For example, a function pointer is a valid argument or return type for a function in most languages, even in C. But almost no protocol schemas support this. In more advanced languages, objects, interfaces, abstract data types, types themselves, type constructors, references to other modules, and many other constructs are valid arguments and return types. You can't express any of that in protocol schemas; this makes many tasks more complicated.
So why do people ever write servers instead of libraries?
A library can do anything that a standalone server can do. But our modern Unix-derived programming environments are slanted towards standalone servers, so some things are easier with standalone servers; hence, servers are almost always the first choice.
Nevertheless, contrary to popular belief, libraries can do anything, including:
The precise details of how to maintain a backwards-compatible signature is language-specific, but it's possible in most languages. For example, don't add a new mandatory argument to an existing function; instead, add an optional argument with a default.
In fact, a library is strictly more powerful than a protocol schema in this regard: If you really wanted to, you could use a protocol schema directly to define the format of the data passed in and out of the library.
This field of techniques is called dynamic software updating. These techniques are actually quite easy if the library only does things that one could do over a protocol schema. If the library uses more than that bare minimum of features, dynamic software updating becomes harder, but still possible.
One interesting use of this ability is to implement extremely high-performance, but backwards-incompatible, wire protocols.
The most common way to support cross-language interaction in libraries is to go through the C type system; C functions can be called from, or implemented in, any language with a C FFI. Going through the C type system can be constraining, so there are many projects which allow one language to call functions in another language without going through C. This also allows a signature in one language to have multiple implementations in other languages.
Obviously: You can run library functions on multiple threads. Different threads can even have different IO and CPU priority levels.
"Software fault isolation" is all about this; SFI allows a library to be run in the same address space without the ability to interfere with other memory in that address space. Other techniques in this vein are used all the time for emulation and virtualization.
You might be concerned about using such fancy techniques. Sometimes, they aren't necessary, because the whole program can safely be given access to the supposedly-privileged resource, perhaps because the resource is one more of:
Consider this carefully; this point is often non-obvious.