SQLite file has a well known and constant header. To determine if file is encrypted (and if password is right or not), just check the header. If it is plain, the file is not encrypted. If after decryption it is what it should be, then password is correct. Otherwise password was wrong.
Alternatively, use a HMAC in addition to just encrypting. If this hash (which takes as is input the block to hash as well as the password or a derivative of it) matches, you know both that was encrypted by you, and that the block has the content you expect it to have.
That's also a cheap way to know if you can open the file or not, without having to decrypt it. If your encryption scheme is such that you sign each encrypted block with a HMAC, then you check the signature before decrypting the block in question. When the block in question contains the header, this gives you an early out.
Either the hash matches and you can decrypt an open the database, or the hash is wrong, meaning someone/something corrupted the database or the wrong password was supplied. You're going to want to sign each ciphertext block anyway, if for no other reason than to make sure you don't end up with garbage after decryption (unless you encrypted garbage by design ;-) ).