I think my initial reaction is to just refuse those special values, but I'll look into the parsing code to see what can be done.
this is not what we expected? For the VALUE part of pg_ndistinct, float8 has 3 special values: inf, -inf, NaN.
For the key part of pg_ndistinct, see example. select '{"1, 16\t":"1"}'::pg_ndistinct; here \t is not tab character, ascii 9. it's two characters: backslash and character "t". so here it should error out? (apply this to \n, \r, \b)
I don't have a good answer as to what should happen here. Special cases like this make Tomas' suggestion to change the in/out format more attractive.
pg_ndistinct_in(PG_FUNCTION_ARGS) ending part should be:
freeJsonLexContext(lex); if (result == JSON_SUCCESS) { ...... } else { ereturn(parse_state.escontext, (Datum) 0, errcode(ERRCODE_INVALID_TEXT_REPRESENTATION), errmsg("malformed pg_ndistinct: \"%s\"", str), errdetail("Must be valid JSON.")); PG_RETURN_NULL(); } result should be either JSON_SUCCESS or anything else.
all these functions: ndistinct_object_start, ndistinct_array_start, ndistinct_object_field_start, ndistinct_array_element_start have ndistinctParseState *parse = state;
do we need to change it to ndistinctParseState *parse = (ndistinctParseState *)state; ?
The compiler isn't complaining so far, but I see no harm in it.
