SSH public keys are encoded in a special binary format, typically represented to end users as either one-liner OpenSSH format or multi-line PEM format (commerical SSH). Text wrapper carries extra information about user who created the key, comment, etc, but the inner binary is always base64-encoded and follows the same internal format.
This format spec deals with this internal binary format (called "blob" in openssh sources) only. Buffer is expected to be raw binary and not base64-d. Implementation closely follows code in OpenSSH.
This page hosts a formal specification of SSH public key using Kaitai Struct. This specification can be automatically translated into a variety of programming languages to get a parsing library.